[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:03:47 dist-test-slave-7kzw krb5kdc[13914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:03:47 dist-test-slave-7kzw krb5kdc[13914](info): set up 2 sockets
Aug 06 16:03:47 dist-test-slave-7kzw krb5kdc[13914](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 16:03:48.946983 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:03:49 dist-test-slave-7kzw krb5kdc[13914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:03:49.099086 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:32843
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:32843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:49.232357 13935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:49.232764 13935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:49.232851 13935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:49.237334 13935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:03:49.237461 13935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:49.237499 13935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:49.237540 13935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:03:49.237581 13935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:03:49.244892 13935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:32843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:32843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:49.246495 13935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:49.247807 13935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:03:49.258590 13935 server_base.cc:1034] running on GCE node
W20260806 16:03:49.258548 13941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:49.258625 13942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:49.258590 13944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:49.259485 13935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:49.261111 13935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:49.262354 13935 hybrid_clock.cc:648] HybridClock initialized: now 1786032229262307 us; error 93 us; skew 500 ppm
Aug 06 16:03:49 dist-test-slave-7kzw krb5kdc[13914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032229, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:49.265852 13935 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:03:49.267012 13935 webserver.cc:460] Webserver started at http://127.13.148.62:34045/ using document root <none> and password file <none>
I20260806 16:03:49.267735 13935 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:49.267840 13935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:49.268209 13935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:49.270407 13935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "ecda73df594e4b36b5a3255865ef19b6"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "55775b7a10420c95c90df14dfb6512f4"
server_key_iv: "3df3f81ba58189d3d37aa80f133caae8"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.271059 13935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "ecda73df594e4b36b5a3255865ef19b6"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "55775b7a10420c95c90df14dfb6512f4"
server_key_iv: "3df3f81ba58189d3d37aa80f133caae8"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.275439 13935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:03:49.278378 13954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:49.279861 13935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:49.280020 13935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "ecda73df594e4b36b5a3255865ef19b6"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "55775b7a10420c95c90df14dfb6512f4"
server_key_iv: "3df3f81ba58189d3d37aa80f133caae8"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.280150 13935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:49.303126 13935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:49.306977 13935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:49.307225 13935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:49.315788 13935 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:32843
I20260806 16:03:49.315814 14027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:32843 every 8 connection(s)
I20260806 16:03:49.316922 13935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:03:49.318894 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 13935
I20260806 16:03:49.319080 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:03:49.320761 14028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:49.320935 13904 external_mini_cluster.cc:1428] Setting key 7f5d71503a6826bfe327db67d14f38de
I20260806 16:03:49.328517 14028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6: Bootstrap starting.
I20260806 16:03:49.331291 14028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:49.332284 14028 log.cc:824] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:49.334901 14028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6: No bootstrap required, opened a new log
I20260806 16:03:49.339582 14028 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecda73df594e4b36b5a3255865ef19b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32843 } }
I20260806 16:03:49.339852 14028 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:49.339941 14028 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecda73df594e4b36b5a3255865ef19b6, State: Initialized, Role: FOLLOWER
I20260806 16:03:49.340471 14028 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecda73df594e4b36b5a3255865ef19b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32843 } }
I20260806 16:03:49.340652 14028 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:03:49.340749 14028 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:03:49.340862 14028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:49.342135 14028 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecda73df594e4b36b5a3255865ef19b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32843 } }
Aug 06 16:03:49 dist-test-slave-7kzw krb5kdc[13914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:49.342562 14028 leader_election.cc:302] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecda73df594e4b36b5a3255865ef19b6; no voters:
I20260806 16:03:49.342975 14028 leader_election.cc:288] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:03:49.343129 14035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:49.343416 14035 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [term 1 LEADER]: Becoming Leader. State: Replica: ecda73df594e4b36b5a3255865ef19b6, State: Running, Role: LEADER
I20260806 16:03:49.343771 14035 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecda73df594e4b36b5a3255865ef19b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32843 } }
I20260806 16:03:49.344467 14028 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:03:49.345995 14036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecda73df594e4b36b5a3255865ef19b6. Latest consensus state: current_term: 1 leader_uuid: "ecda73df594e4b36b5a3255865ef19b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecda73df594e4b36b5a3255865ef19b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32843 } } }
I20260806 16:03:49.346161 14036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:03:49.345990 14037 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecda73df594e4b36b5a3255865ef19b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecda73df594e4b36b5a3255865ef19b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32843 } } }
I20260806 16:03:49.346273 14037 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:03:49.346737 14048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:03:49.350750 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:49.325261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.13.148.62:32843)
0806 16:03:49.325791 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:49.325812 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:49.326850 (+ 1038us) server_negotiation.cc:407] Connection header received
0806 16:03:49.330989 (+ 4139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:49.331008 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:49.331388 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:49.331695 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:49.334672 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.335734 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.337438 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.337919 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.338044 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:49.343661 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:49.343669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:49.343685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:49.343722 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:49.346144 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:49.347235 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:49.347244 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:49.347249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:49.347416 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:49.347896 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:49.347900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:49.347902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:49.348471 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:49.348691 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:49.349386 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 16:03:49.349675 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":148,"threads_started":1}
I20260806 16:03:49.350891 14048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:03:49.357736 14048 catalog_manager.cc:1347] Generated new cluster ID: a677e34b4ccb4f659aa071567ac289d9
I20260806 16:03:49.357873 14048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:03:49.370071 14048 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:03:49.371457 14048 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:03:49.386204 14048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecda73df594e4b36b5a3255865ef19b6: Generated new TSK 0
I20260806 16:03:49.387182 14048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:03:49.424383 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32843
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:49.543632 14061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:49.544030 14061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:49.544080 14061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:49.548659 14061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:49.548741 14061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:49.548838 14061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:03:49.554510 14061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:49.555655 14061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:49.556734 14061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:49.566933 14069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:49.566933 14068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:49.567101 14061 server_base.cc:1034] running on GCE node
W20260806 16:03:49.567332 14072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:49.567822 14061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:49.568562 14061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:49.569796 14061 hybrid_clock.cc:648] HybridClock initialized: now 1786032229569778 us; error 29 us; skew 500 ppm
Aug 06 16:03:49 dist-test-slave-7kzw krb5kdc[13914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032229, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:49.573123 14061 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:03:49.574206 14061 webserver.cc:460] Webserver started at http://127.13.148.1:46781/ using document root <none> and password file <none>
I20260806 16:03:49.574947 14061 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:49.575088 14061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:49.575350 14061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:49.577632 14061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "3f7e7c9675c94ae3a1b86bc9a519740a"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "d5a1ff72bf3255d1caf5876a31091a4d"
server_key_iv: "3766f988cb854ae4769401d130824186"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.578289 14061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "3f7e7c9675c94ae3a1b86bc9a519740a"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "d5a1ff72bf3255d1caf5876a31091a4d"
server_key_iv: "3766f988cb854ae4769401d130824186"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.582719 14061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:03:49.585749 14080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:49.587284 14061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:49.587459 14061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "3f7e7c9675c94ae3a1b86bc9a519740a"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "d5a1ff72bf3255d1caf5876a31091a4d"
server_key_iv: "3766f988cb854ae4769401d130824186"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.587606 14061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:49.615538 14061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:49.619361 14061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:49.619618 14061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:49.620304 14061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:03:49.621258 14061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:49.621338 14061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:49.621417 14061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:49.621454 14061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:49.631474 14061 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:38795
I20260806 16:03:49.631493 14236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:38795 every 8 connection(s)
I20260806 16:03:49.632535 14061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:03:49.642521 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 14061
I20260806 16:03:49.642660 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:03:49.642985 13904 external_mini_cluster.cc:1428] Setting key ff8bd55895187ffbe0dfad401b233067
Aug 06 16:03:49 dist-test-slave-7kzw krb5kdc[13914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032229, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:49.648567 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:49.634746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:43597 (local address 127.13.148.62:32843)
0806 16:03:49.634894 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:49.634898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:49.635796 (+ 898us) server_negotiation.cc:407] Connection header received
0806 16:03:49.637387 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:49.637391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:49.637455 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:49.637561 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:49.638890 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.639442 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.640667 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.640916 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.641035 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:49.644110 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:49.644116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:49.644120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:49.644159 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:49.645814 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:49.646560 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:49.646564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:49.646565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:49.646657 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:49.647175 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:49.647181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:49.647186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:49.647456 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:49.647556 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:49.648190 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 16:03:49.648355 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:03:49.649507 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:49.635138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32843 (local address 127.13.148.1:43597)
0806 16:03:49.635618 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:49.635665 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:49.637148 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:49.637755 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:49.637762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:49.638155 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:49.638728 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:49.638751 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.639583 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.639591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:49.640535 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:49.640545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.641083 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.641087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:49.641261 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:49.641908 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:49.641926 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:49.643888 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:49.645943 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:49.645952 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:49.645975 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:49.646387 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:49.646800 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:49.646803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:49.646804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:49.647033 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:49.647620 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:49.647625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:49.647889 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 16:03:49.648085 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 16:03:49.648393 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260806 16:03:49.650950 14237 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32843
I20260806 16:03:49.651211 14237 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:49.651741 14237 heartbeater.cc:511] Master 127.13.148.62:32843 requested a full tablet report, sending...
I20260806 16:03:49.653620 13977 ts_manager.cc:194] Registered new tserver with Master: 3f7e7c9675c94ae3a1b86bc9a519740a (127.13.148.1:38795)
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:03:49.655382 13977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:43597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:03:49.678066 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32843
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:49.800077 14246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:49.800410 14246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:49.800457 14246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:49.804836 14246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:49.804929 14246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:49.804996 14246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:03:49.810916 14246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:49.812158 14246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:49.813271 14246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:49.824165 14252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:49.824177 14251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:49.824165 14254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:49.824581 14246 server_base.cc:1034] running on GCE node
I20260806 16:03:49.825093 14246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:49.825868 14246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:49.827096 14246 hybrid_clock.cc:648] HybridClock initialized: now 1786032229827047 us; error 62 us; skew 500 ppm
Aug 06 16:03:49 dist-test-slave-7kzw krb5kdc[13914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032229, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:49.830443 14246 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:03:49.831624 14246 webserver.cc:460] Webserver started at http://127.13.148.2:43209/ using document root <none> and password file <none>
I20260806 16:03:49.832289 14246 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:49.832399 14246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:49.832743 14246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:49.835254 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "4b6c275d6e774689845a2e9076aba677"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "61fd4a9cbc817859a86072275241ecce"
server_key_iv: "942cd1a7b44802a0cb7a3447ee288785"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.835911 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "4b6c275d6e774689845a2e9076aba677"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "61fd4a9cbc817859a86072275241ecce"
server_key_iv: "942cd1a7b44802a0cb7a3447ee288785"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.840322 14246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:03:49.843235 14261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:49.844762 14246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:49.844905 14246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "4b6c275d6e774689845a2e9076aba677"
format_stamp: "Formatted at 2026-08-06 16:03:49 on dist-test-slave-7kzw"
server_key: "61fd4a9cbc817859a86072275241ecce"
server_key_iv: "942cd1a7b44802a0cb7a3447ee288785"
server_key_version: "encryptionkey@0"
I20260806 16:03:49.845052 14246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:49.891781 14246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:49.895776 14246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:49.896061 14246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:49.896842 14246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:03:49.897866 14246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:49.897946 14246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:49.898028 14246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:49.898085 14246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:49.912199 14246 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:40047
I20260806 16:03:49.912222 14425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:40047 every 8 connection(s)
I20260806 16:03:49.913257 14246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:03:49.917657 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 14246
I20260806 16:03:49.917814 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:03:49.918067 13904 external_mini_cluster.cc:1428] Setting key 4bd760b696ab5273824a580d786bc6e4
Aug 06 16:03:49 dist-test-slave-7kzw krb5kdc[13914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032229, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:03:49.930609 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:49.915657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:58831 (local address 127.13.148.62:32843)
0806 16:03:49.915815 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:49.915819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:49.916817 (+ 998us) server_negotiation.cc:407] Connection header received
0806 16:03:49.918331 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:49.918334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:49.918395 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:49.918520 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:49.920196 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.920747 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.922454 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.922919 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.923049 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:49.926134 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:49.926137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:49.926139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:49.926166 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:49.927895 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:49.928634 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:49.928637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:49.928639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:49.928725 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:49.929182 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:49.929185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:49.929186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:49.929400 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:49.929482 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:49.930228 (+ 746us) server_negotiation.cc:299] Negotiation successful
0806 16:03:49.930417 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:03:49.931602 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:49.915940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32843 (local address 127.13.148.2:58831)
0806 16:03:49.916506 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:49.916552 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:49.918054 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:49.918747 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:49.918755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:49.919311 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:49.919976 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:49.919998 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.920900 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.920907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:49.922310 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:49.922321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:49.923078 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:49.923081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:49.923254 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:49.923935 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:49.923953 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:49.925930 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:49.928029 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:49.928040 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:49.928061 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:49.928484 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:49.928845 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:49.928849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:49.928851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:49.929061 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:49.929543 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:49.929553 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:49.929931 (+ 378us) client_negotiation.cc:769] Sending connection context
0806 16:03:49.930154 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 16:03:49.930605 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":187,"threads_started":1}
I20260806 16:03:49.933022 14426 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32843
I20260806 16:03:49.933382 14426 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260806 16:03:49.934000 14426 heartbeater.cc:511] Master 127.13.148.62:32843 requested a full tablet report, sending...
I20260806 16:03:49.935621 13977 ts_manager.cc:194] Registered new tserver with Master: 4b6c275d6e774689845a2e9076aba677 (127.13.148.2:40047)
I20260806 16:03:49.936800 13977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:58831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:03:49.954795 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32843
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:50.077047 14433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:50.077385 14433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:50.077435 14433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:50.081825 14433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:50.081913 14433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:50.081979 14433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:03:50.087854 14433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:50.088960 14433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:50.089991 14433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:50.100517 14441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:50.100530 14438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:50.100517 14439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:50.100606 14433 server_base.cc:1034] running on GCE node
I20260806 16:03:50.101284 14433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:50.102108 14433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:50.103330 14433 hybrid_clock.cc:648] HybridClock initialized: now 1786032230103269 us; error 74 us; skew 500 ppm
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[13914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032230, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:50.106382 14433 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:03:50.107430 14433 webserver.cc:460] Webserver started at http://127.13.148.3:45257/ using document root <none> and password file <none>
I20260806 16:03:50.108086 14433 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:50.108199 14433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:50.108454 14433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:50.110678 14433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "d6a0f65ea8fd42118be2b6ea113b9924"
format_stamp: "Formatted at 2026-08-06 16:03:50 on dist-test-slave-7kzw"
server_key: "2bf424a736e0f2075b7c832b854da07b"
server_key_iv: "f5fd6b947ee6754844d5f3a3314e3895"
server_key_version: "encryptionkey@0"
I20260806 16:03:50.111274 14433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "d6a0f65ea8fd42118be2b6ea113b9924"
format_stamp: "Formatted at 2026-08-06 16:03:50 on dist-test-slave-7kzw"
server_key: "2bf424a736e0f2075b7c832b854da07b"
server_key_iv: "f5fd6b947ee6754844d5f3a3314e3895"
server_key_version: "encryptionkey@0"
I20260806 16:03:50.115417 14433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:03:50.118136 14453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:50.119725 14433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:03:50.119908 14433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "d6a0f65ea8fd42118be2b6ea113b9924"
format_stamp: "Formatted at 2026-08-06 16:03:50 on dist-test-slave-7kzw"
server_key: "2bf424a736e0f2075b7c832b854da07b"
server_key_iv: "f5fd6b947ee6754844d5f3a3314e3895"
server_key_version: "encryptionkey@0"
I20260806 16:03:50.120059 14433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:50.136363 14433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:50.140759 14433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:50.141043 14433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:50.141748 14433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:03:50.142831 14433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:50.142908 14433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:50.142985 14433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:50.143039 14433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:50.160075 14433 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:41445
I20260806 16:03:50.160100 14610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:41445 every 8 connection(s)
I20260806 16:03:50.161257 14433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:03:50.164474 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 14433
I20260806 16:03:50.164618 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:03:50.164920 13904 external_mini_cluster.cc:1428] Setting key 01de0e8d1ccad82d7156a901af678a51
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[13914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032230, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:50.179477 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:50.164308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:38399 (local address 127.13.148.62:32843)
0806 16:03:50.164468 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:50.164474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:50.165690 (+ 1216us) server_negotiation.cc:407] Connection header received
0806 16:03:50.167337 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:50.167341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:50.167402 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:50.167492 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:50.168867 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:50.169477 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:50.170927 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:50.171217 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:50.171352 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:50.174459 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:50.174522 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:50.174528 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:50.174565 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:50.177055 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:50.177685 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:50.177689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:50.177691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:50.177795 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:50.178191 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:50.178194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:50.178196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:50.178413 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:50.178584 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:50.179166 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 16:03:50.179307 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:03:50.180471 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:50.164820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32843 (local address 127.13.148.3:38399)
0806 16:03:50.165496 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:50.165565 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:50.167069 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:50.167697 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:50.167706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:50.168131 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:50.168692 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:50.168709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:50.169642 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:50.169646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:50.170783 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:50.170794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:50.171406 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:50.171411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:50.171610 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:50.172349 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:50.172368 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:50.174213 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:50.177219 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:50.177227 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:50.177244 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:50.177557 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:50.177900 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:50.177903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:50.177908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:50.178079 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:50.178583 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:50.178593 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:50.178887 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 16:03:50.179098 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 16:03:50.179495 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":282,"threads_started":1}
I20260806 16:03:50.181743 14612 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32843
I20260806 16:03:50.182054 14612 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:50.182674 14612 heartbeater.cc:511] Master 127.13.148.62:32843 requested a full tablet report, sending...
I20260806 16:03:50.184268 13977 ts_manager.cc:194] Registered new tserver with Master: d6a0f65ea8fd42118be2b6ea113b9924 (127.13.148.3:41445)
I20260806 16:03:50.184924 13977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:38399
I20260806 16:03:50.192653 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[13914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[13914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:03:50.227907 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:50.216653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.13.148.1:38795)
0806 16:03:50.216929 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:50.216940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:50.216980 (+ 40us) server_negotiation.cc:407] Connection header received
0806 16:03:50.217113 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:50.217119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:50.217329 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:50.217471 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:50.218401 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:50.219376 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:50.220449 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:50.220861 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:50.220951 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:50.223704 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:50.223715 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:50.223735 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:50.223765 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:50.225753 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:50.226255 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:50.226261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:50.226262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:50.226359 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:50.226774 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:50.226777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:50.226779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:50.227136 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:50.227356 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:50.227571 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 16:03:50.227739 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
W20260806 16:03:50.229067 14121 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36110
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[13914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[13914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:03:50.264076 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:50.254567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.13.148.1:38795)
0806 16:03:50.254747 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:50.254751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:50.254905 (+ 154us) server_negotiation.cc:407] Connection header received
0806 16:03:50.255034 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:50.255037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:50.255083 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:50.255167 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:50.255975 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:50.256603 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:50.257509 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:50.257763 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:50.257857 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:50.260604 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:50.260607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:50.260609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:50.260639 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:50.262216 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:50.262762 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:50.262766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:50.262768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:50.262857 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:50.263167 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:50.263170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:50.263172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:50.263362 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:50.263511 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:50.263767 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 16:03:50.263896 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:03:50.265255 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 14061
I20260806 16:03:50.279947 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 14246
I20260806 16:03:50.292940 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 14433
I20260806 16:03:50.307130 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 13935
2026-08-06T16:03:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2703 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[14630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[14630](info): set up 2 sockets
Aug 06 16:03:50 dist-test-slave-7kzw krb5kdc[14630](info): commencing operation
krb5kdc: starting...
W20260806 16:03:51.574841 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:03:51 dist-test-slave-7kzw krb5kdc[14630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:03:51.722726 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40809
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:51.865116 14647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:51.865433 14647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:51.865477 14647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:51.869664 14647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:03:51.869735 14647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:51.869755 14647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:51.869772 14647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:03:51.869788 14647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:03:51.875455 14647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:51.876551 14647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:51.877568 14647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:51.886899 14662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:51.886973 14656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:51.886899 14654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:51.887171 14647 server_base.cc:1034] running on GCE node
I20260806 16:03:51.887889 14647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:51.889274 14647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:51.890560 14647 hybrid_clock.cc:648] HybridClock initialized: now 1786032231890499 us; error 75 us; skew 500 ppm
Aug 06 16:03:51 dist-test-slave-7kzw krb5kdc[14630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032231, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:51.893716 14647 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:03:51.894784 14647 webserver.cc:460] Webserver started at http://127.13.148.62:34873/ using document root <none> and password file <none>
I20260806 16:03:51.895401 14647 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:51.895498 14647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:51.895753 14647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:51.897923 14647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "d2abf68d79584e0a97849472c5ce5786"
format_stamp: "Formatted at 2026-08-06 16:03:51 on dist-test-slave-7kzw"
server_key: "608dd94cd729da43b0fc511214784ebe"
server_key_iv: "8ab754f334810391f18009aa250f30d9"
server_key_version: "encryptionkey@0"
I20260806 16:03:51.898619 14647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "d2abf68d79584e0a97849472c5ce5786"
format_stamp: "Formatted at 2026-08-06 16:03:51 on dist-test-slave-7kzw"
server_key: "608dd94cd729da43b0fc511214784ebe"
server_key_iv: "8ab754f334810391f18009aa250f30d9"
server_key_version: "encryptionkey@0"
I20260806 16:03:51.902874 14647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:03:51.905757 14674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:51.907409 14647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:51.907612 14647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "d2abf68d79584e0a97849472c5ce5786"
format_stamp: "Formatted at 2026-08-06 16:03:51 on dist-test-slave-7kzw"
server_key: "608dd94cd729da43b0fc511214784ebe"
server_key_iv: "8ab754f334810391f18009aa250f30d9"
server_key_version: "encryptionkey@0"
I20260806 16:03:51.907770 14647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:51.937104 14647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:51.940514 14647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:51.940764 14647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:51.948710 14647 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40809
I20260806 16:03:51.948725 14748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40809 every 8 connection(s)
I20260806 16:03:51.949775 14647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:03:51.952024 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 14647
I20260806 16:03:51.952162 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:03:51.952466 13904 external_mini_cluster.cc:1428] Setting key 4aa7f366fd03f0699ad67b383e526494
I20260806 16:03:51.953377 14749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:51.961133 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786: Bootstrap starting.
Aug 06 16:03:51 dist-test-slave-7kzw krb5kdc[14630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:51.964464 14749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:51.965585 14749 log.cc:824] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:51.968501 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786: No bootstrap required, opened a new log
I20260806 16:03:51.969828 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:51.954072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.13.148.62:40809)
0806 16:03:51.954538 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:51.954561 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:51.954609 (+ 48us) server_negotiation.cc:407] Connection header received
0806 16:03:51.955257 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:51.955274 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:51.955601 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:51.955887 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:51.956980 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:51.958089 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:51.959434 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:51.959971 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:51.960118 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:51.963597 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:51.963605 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:51.963626 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:51.963679 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:51.966161 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:51.966853 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:51.966861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:51.966868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:51.967017 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:51.967344 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:51.967348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:51.967349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:51.967929 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:51.968154 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:51.968493 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 16:03:51.968844 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":145,"threads_started":1}
I20260806 16:03:51.972484 14749 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2abf68d79584e0a97849472c5ce5786" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40809 } }
I20260806 16:03:51.972802 14749 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:51.972869 14749 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2abf68d79584e0a97849472c5ce5786, State: Initialized, Role: FOLLOWER
I20260806 16:03:51.973282 14749 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2abf68d79584e0a97849472c5ce5786" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40809 } }
I20260806 16:03:51.973403 14749 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:03:51.973443 14749 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:03:51.973515 14749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:51.974438 14749 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2abf68d79584e0a97849472c5ce5786" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40809 } }
I20260806 16:03:51.974818 14749 leader_election.cc:302] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2abf68d79584e0a97849472c5ce5786; no voters:
I20260806 16:03:51.975168 14749 leader_election.cc:288] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:03:51.975217 14757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:51.975425 14757 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [term 1 LEADER]: Becoming Leader. State: Replica: d2abf68d79584e0a97849472c5ce5786, State: Running, Role: LEADER
I20260806 16:03:51.975749 14757 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2abf68d79584e0a97849472c5ce5786" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40809 } }
I20260806 16:03:51.976522 14749 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:03:51.977669 14761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2abf68d79584e0a97849472c5ce5786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2abf68d79584e0a97849472c5ce5786" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40809 } } }
I20260806 16:03:51.977705 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2abf68d79584e0a97849472c5ce5786. Latest consensus state: current_term: 1 leader_uuid: "d2abf68d79584e0a97849472c5ce5786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2abf68d79584e0a97849472c5ce5786" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40809 } } }
I20260806 16:03:51.977828 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [sys.catalog]: This master's current role is: LEADER
I20260806 16:03:51.977830 14761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786 [sys.catalog]: This master's current role is: LEADER
I20260806 16:03:51.978259 14771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:03:51.981618 14771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:03:51.987332 14771 catalog_manager.cc:1347] Generated new cluster ID: 9b241dff915543db88f411c5965543fe
I20260806 16:03:51.987416 14771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:03:52.002643 14771 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:03:52.003989 14771 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:03:52.011371 14771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2abf68d79584e0a97849472c5ce5786: Generated new TSK 0
I20260806 16:03:52.012239 14771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:03:52.046280 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40809
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:52.170903 14787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:52.171231 14787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:52.171279 14787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:52.175624 14787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:52.175712 14787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:52.175776 14787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:03:52.181597 14787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:52.182843 14787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:52.183985 14787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:52.193994 14796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:52.193994 14797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:52.194039 14800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:52.194288 14787 server_base.cc:1034] running on GCE node
I20260806 16:03:52.194787 14787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:52.195425 14787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:52.196589 14787 hybrid_clock.cc:648] HybridClock initialized: now 1786032232196565 us; error 39 us; skew 500 ppm
Aug 06 16:03:52 dist-test-slave-7kzw krb5kdc[14630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032232, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:52.199888 14787 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:03:52.200903 14787 webserver.cc:460] Webserver started at http://127.13.148.1:33599/ using document root <none> and password file <none>
I20260806 16:03:52.201562 14787 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:52.201668 14787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:52.201942 14787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:52.204355 14787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "1b4de99c21170225edf080ddf8a9ca48"
server_key_iv: "c32be78151a6cbd83555d7337cdfaa28"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.205015 14787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "1b4de99c21170225edf080ddf8a9ca48"
server_key_iv: "c32be78151a6cbd83555d7337cdfaa28"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.209422 14787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 16:03:52.212314 14811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.213740 14787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:03:52.213908 14787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "1b4de99c21170225edf080ddf8a9ca48"
server_key_iv: "c32be78151a6cbd83555d7337cdfaa28"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.214039 14787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:52.246829 14787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:52.250681 14787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:52.250942 14787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:52.251693 14787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:03:52.252614 14787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:52.252689 14787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.252763 14787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:52.252812 14787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.261813 14787 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:36037
I20260806 16:03:52.261847 14964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:36037 every 8 connection(s)
I20260806 16:03:52.262907 14787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:03:52.265161 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 14787
I20260806 16:03:52.265328 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:03:52.265630 13904 external_mini_cluster.cc:1428] Setting key 3167c3b60b3d280fc7daaaf7d283e062
Aug 06 16:03:52 dist-test-slave-7kzw krb5kdc[14630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032232, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:03:52.278369 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.264948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:54591 (local address 127.13.148.62:40809)
0806 16:03:52.265154 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:52.265161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:52.266056 (+ 895us) server_negotiation.cc:407] Connection header received
0806 16:03:52.267696 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:52.267700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:52.267754 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:52.267835 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:52.269067 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.269631 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.270899 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.271152 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.271276 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.274304 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:52.274308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:52.274310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:52.274336 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:52.276000 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:52.276590 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:52.276597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:52.276599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:52.276690 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:52.277048 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:52.277080 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:52.277084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:52.277311 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:52.277430 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:52.278035 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 16:03:52.278177 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 16:03:52.279337 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.265335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40809 (local address 127.13.148.1:54591)
0806 16:03:52.265892 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:52.265950 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:52.267470 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:52.268020 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:52.268027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:52.268404 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:52.268909 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.268924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.269775 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.269778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.270771 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.270782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.271327 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.271331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.271506 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.272234 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:52.272251 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:52.274100 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:52.276125 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:52.276132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:52.276151 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:52.276434 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:52.276793 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:52.276796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:52.276798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:52.276948 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:52.277428 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:52.277435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:52.277787 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 16:03:52.278000 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 16:03:52.278393 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":152,"threads_started":1}
I20260806 16:03:52.280967 14965 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40809
I20260806 16:03:52.281281 14965 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:03:52.281947 14965 heartbeater.cc:511] Master 127.13.148.62:40809 requested a full tablet report, sending...
I20260806 16:03:52.284073 14695 ts_manager.cc:194] Registered new tserver with Master: 40fcceeb71cf4f6c9cf5cef9ba1e3d3a (127.13.148.1:36037)
I20260806 16:03:52.285318 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:54591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:03:52.302937 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40809
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:52.422269 14970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:52.422627 14970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:52.422709 14970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:52.426795 14970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:52.426859 14970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:52.426949 14970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:03:52.432327 14970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:52.433459 14970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:52.434553 14970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:52.444386 14985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:52.444386 14984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:52.444386 14989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:52.444854 14970 server_base.cc:1034] running on GCE node
I20260806 16:03:52.445251 14970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:52.445905 14970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:52.447115 14970 hybrid_clock.cc:648] HybridClock initialized: now 1786032232447092 us; error 33 us; skew 500 ppm
Aug 06 16:03:52 dist-test-slave-7kzw krb5kdc[14630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032232, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:52.450420 14970 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:03:52.451581 14970 webserver.cc:460] Webserver started at http://127.13.148.2:37361/ using document root <none> and password file <none>
I20260806 16:03:52.452245 14970 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:52.452328 14970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:52.452580 14970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:52.454854 14970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "b43f14b585e2433882dc3b1ca6a08699"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "8a44490ff2f50dbe8b4c8a9caac29a7c"
server_key_iv: "a82a5faf1f57c663feed569798f1dffe"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.455536 14970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "b43f14b585e2433882dc3b1ca6a08699"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "8a44490ff2f50dbe8b4c8a9caac29a7c"
server_key_iv: "a82a5faf1f57c663feed569798f1dffe"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.459698 14970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:03:52.462322 14997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.463586 14970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:03:52.463761 14970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "b43f14b585e2433882dc3b1ca6a08699"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "8a44490ff2f50dbe8b4c8a9caac29a7c"
server_key_iv: "a82a5faf1f57c663feed569798f1dffe"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.463881 14970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:52.493402 14970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:52.497051 14970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:52.497275 14970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:52.497929 14970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:03:52.498885 14970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:52.498947 14970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.498992 14970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:52.499008 14970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.509050 14970 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:37645
I20260806 16:03:52.509074 15163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:37645 every 8 connection(s)
I20260806 16:03:52.510035 14970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:03:52.511413 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 14970
I20260806 16:03:52.511543 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:03:52.511847 13904 external_mini_cluster.cc:1428] Setting key a06e6325d8df2794a166a0b680e8b056
Aug 06 16:03:52 dist-test-slave-7kzw krb5kdc[14630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032232, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:03:52.525942 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.512244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:48165 (local address 127.13.148.62:40809)
0806 16:03:52.512390 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:52.512396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:52.513449 (+ 1053us) server_negotiation.cc:407] Connection header received
0806 16:03:52.514367 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:52.514371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:52.514425 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:52.514541 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:52.516144 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.517017 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.518300 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.518767 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.518889 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.521790 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:52.521795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:52.521800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:52.521837 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:52.523636 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:52.524267 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:52.524270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:52.524271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:52.524353 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:52.524785 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:52.524788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:52.524794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:52.524992 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:52.525068 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:52.525660 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 16:03:52.525813 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260806 16:03:52.527186 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.512746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40809 (local address 127.13.148.2:48165)
0806 16:03:52.513286 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:52.513346 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:52.514163 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:52.514734 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:52.514746 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:52.515301 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:52.515984 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.516000 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.517149 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.517152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.518174 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.518184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.518959 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.518964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.519138 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.519815 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:52.519832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:52.521596 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:52.523802 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:52.523810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:52.523830 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:52.524154 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:52.524456 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:52.524458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:52.524460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:52.524617 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:52.525110 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:52.525117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:52.525394 (+ 277us) client_negotiation.cc:769] Sending connection context
0806 16:03:52.525587 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 16:03:52.526000 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
I20260806 16:03:52.528648 15165 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40809
I20260806 16:03:52.528925 15165 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:52.529471 15165 heartbeater.cc:511] Master 127.13.148.62:40809 requested a full tablet report, sending...
I20260806 16:03:52.530642 14695 ts_manager.cc:194] Registered new tserver with Master: b43f14b585e2433882dc3b1ca6a08699 (127.13.148.2:37645)
I20260806 16:03:52.531508 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:48165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:03:52.546139 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40809
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:52.663597 15171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:52.663987 15171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:52.664080 15171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:52.669203 15171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:52.669296 15171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:52.669370 15171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:03:52.675781 15171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:52.677093 15171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:52.678392 15171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:03:52.689584 15171 server_base.cc:1034] running on GCE node
W20260806 16:03:52.689518 15180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:52.689518 15178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:52.689512 15182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:52.690358 15171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:52.691087 15171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:52.692265 15171 hybrid_clock.cc:648] HybridClock initialized: now 1786032232692227 us; error 50 us; skew 500 ppm
Aug 06 16:03:52 dist-test-slave-7kzw krb5kdc[14630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032232, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:52.698976 15171 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:03:52.700011 15171 webserver.cc:460] Webserver started at http://127.13.148.3:38505/ using document root <none> and password file <none>
I20260806 16:03:52.700670 15171 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:52.700774 15171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:52.701031 15171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:52.703334 15171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "81921810119e481a9c2352fe329433dc"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "1ef0444cec6fef9b59a5c64e5cc177b6"
server_key_iv: "e0878e55de966693055965ea09c506bf"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.703954 15171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "81921810119e481a9c2352fe329433dc"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "1ef0444cec6fef9b59a5c64e5cc177b6"
server_key_iv: "e0878e55de966693055965ea09c506bf"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.708166 15171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:03:52.711153 15196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.712821 15171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:03:52.712971 15171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "81921810119e481a9c2352fe329433dc"
format_stamp: "Formatted at 2026-08-06 16:03:52 on dist-test-slave-7kzw"
server_key: "1ef0444cec6fef9b59a5c64e5cc177b6"
server_key_iv: "e0878e55de966693055965ea09c506bf"
server_key_version: "encryptionkey@0"
I20260806 16:03:52.713099 15171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:52.745337 15171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:52.749189 15171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:52.749467 15171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:52.750240 15171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:03:52.751305 15171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:52.751396 15171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.751489 15171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:52.751588 15171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:52.762673 15171 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:37263
I20260806 16:03:52.762694 15362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:37263 every 8 connection(s)
I20260806 16:03:52.763720 15171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:03:52.765023 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 15171
I20260806 16:03:52.765179 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:03:52.765564 13904 external_mini_cluster.cc:1428] Setting key 34da6e66c645c5b1738fec6476eb5d9c
Aug 06 16:03:52 dist-test-slave-7kzw krb5kdc[14630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032232, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:52.778959 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.765921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:60893 (local address 127.13.148.62:40809)
0806 16:03:52.766091 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:52.766097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:52.767214 (+ 1117us) server_negotiation.cc:407] Connection header received
0806 16:03:52.768829 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:52.768831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:52.768880 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:52.768954 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:52.770160 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.770718 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.771893 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.772109 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.772196 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.774911 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:52.774914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:52.774917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:52.774964 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:52.776387 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:52.776986 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:52.776989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:52.776990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:52.777102 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:52.777495 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:52.777498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:52.777499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:52.777705 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:52.777783 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:52.778341 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 16:03:52.778448 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:03:52.779981 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.766405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40809 (local address 127.13.148.3:60893)
0806 16:03:52.767050 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:52.767101 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:52.768615 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:52.769129 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:52.769136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:52.769507 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:52.770022 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.770035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.770854 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.770857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.771765 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.771774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.772259 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.772262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.772409 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.773044 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:52.773059 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:52.774731 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:52.776511 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:52.776517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:52.776531 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:52.776892 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:52.777209 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:52.777211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:52.777213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:52.777386 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:52.777814 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:52.777820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:52.778090 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 16:03:52.778311 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 16:03:52.778747 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":182,"threads_started":1}
I20260806 16:03:52.781461 15366 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40809
I20260806 16:03:52.781759 15366 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:52.782313 15366 heartbeater.cc:511] Master 127.13.148.62:40809 requested a full tablet report, sending...
I20260806 16:03:52.783506 14695 ts_manager.cc:194] Registered new tserver with Master: 81921810119e481a9c2352fe329433dc (127.13.148.3:37263)
I20260806 16:03:52.784122 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:60893
I20260806 16:03:52.792896 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:03:52.805529 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.796836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.13.148.62:40809)
0806 16:03:52.796996 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:52.797000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:52.797015 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:03:52.797162 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:52.797165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:52.797210 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:52.797288 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:52.798060 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.798736 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.799808 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.800036 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.800135 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.801826 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:52.801830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:52.801831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:52.801858 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:52.803488 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:52.804013 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:52.804016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:52.804018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:52.804114 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:52.804464 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:52.804468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:52.804469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:52.804764 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:52.804887 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:52.805133 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 16:03:52.805247 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 16:03:52.811694 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:03:52.815317 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:03:52.832118 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.826690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.13.148.2:37645)
0806 16:03:52.827096 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:52.827108 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:52.827133 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:03:52.827208 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:52.827213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:52.827413 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:52.827589 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:52.827990 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.828879 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.830423 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.831330 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.831461 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.831619 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:52.831825 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 16:03:52.831941 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":129,"threads_started":1}
I20260806 16:03:52.832233 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.825684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37645 (local address 127.0.0.1:58896)
0806 16:03:52.826429 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:52.826443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:52.826547 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:52.827750 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:52.827753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:52.827775 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:52.827869 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.827873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.829016 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.829020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.830285 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.830295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.831472 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.831478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.831613 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.831655 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 16:03:52.831772 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 16:03:52.831898 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":28,"thread_start_us":101,"threads_started":1}
I20260806 16:03:52.832180 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.826426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:37263 (local address 127.0.0.1:52490)
0806 16:03:52.827520 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:52.827536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:52.827629 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:52.828145 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:52.828148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:52.828167 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:52.828254 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.828261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.829609 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.829613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.830760 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.830767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.831741 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.831748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.831833 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.831853 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 16:03:52.831916 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 16:03:52.831973 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":35,"thread_start_us":117,"threads_started":1}
I20260806 16:03:52.832600 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.826720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.13.148.3:37263)
0806 16:03:52.827475 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:52.827488 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:52.827619 (+ 131us) server_negotiation.cc:407] Connection header received
0806 16:03:52.827736 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:52.827741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:52.827912 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:52.828145 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:52.828374 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.829447 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.830869 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.831425 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.831532 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.831677 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:52.832088 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 16:03:52.832183 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":98,"threads_started":1}
I20260806 16:03:52.832834 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.825781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.13.148.1:36037)
0806 16:03:52.826245 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:52.826252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:52.826277 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:03:52.826550 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:52.826557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:52.826792 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:52.827080 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:52.827547 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.828775 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.830667 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.831420 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.831509 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.831619 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:52.832418 (+ 799us) server_negotiation.cc:299] Negotiation successful
0806 16:03:52.832537 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":150,"threads_started":1}
I20260806 16:03:52.832170 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:52.825631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:36037 (local address 127.0.0.1:47136)
0806 16:03:52.826094 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:52.826151 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:52.826290 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:52.827174 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:52.827181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:52.827208 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:52.827401 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.827412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.828914 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.828920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.830522 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:52.830532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:52.831702 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:52.831705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:52.831823 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:52.831847 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:03:52.831919 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 16:03:52.831996 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":81,"thread_start_us":106,"threads_started":1}
I20260806 16:03:52.835701 14879 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee36e885f3846b6a47449e05a7a8740 (DEFAULT_TABLE table=test-table [id=b2eeefc7fec64dbda6bc052e88bf9e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:52.835634 15062 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee36e885f3846b6a47449e05a7a8740 (DEFAULT_TABLE table=test-table [id=b2eeefc7fec64dbda6bc052e88bf9e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:52.835634 15263 tablet_service.cc:1467] Processing CreateTablet for tablet 2ee36e885f3846b6a47449e05a7a8740 (DEFAULT_TABLE table=test-table [id=b2eeefc7fec64dbda6bc052e88bf9e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:52.836702 14879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee36e885f3846b6a47449e05a7a8740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:52.836726 15263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee36e885f3846b6a47449e05a7a8740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:52.836721 15062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ee36e885f3846b6a47449e05a7a8740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:52.843137 15389 tablet_bootstrap.cc:492] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a: Bootstrap starting.
I20260806 16:03:52.844662 15392 tablet_bootstrap.cc:492] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699: Bootstrap starting.
I20260806 16:03:52.846233 15389 tablet_bootstrap.cc:654] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:52.846910 15390 tablet_bootstrap.cc:492] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc: Bootstrap starting.
I20260806 16:03:52.846997 15392 tablet_bootstrap.cc:654] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:52.847100 15389 log.cc:824] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:52.847831 15392 log.cc:824] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:52.849121 15390 tablet_bootstrap.cc:654] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:52.849159 15389 tablet_bootstrap.cc:492] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a: No bootstrap required, opened a new log
I20260806 16:03:52.849355 15389 ts_tablet_manager.cc:1397] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:03:52.849932 15392 tablet_bootstrap.cc:492] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699: No bootstrap required, opened a new log
I20260806 16:03:52.850106 15392 ts_tablet_manager.cc:1397] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 16:03:52.850325 15390 log.cc:824] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:52.852425 15390 tablet_bootstrap.cc:492] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc: No bootstrap required, opened a new log
I20260806 16:03:52.852603 15390 ts_tablet_manager.cc:1397] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:03:52.853444 15389 raft_consensus.cc:348] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:52.853698 15389 raft_consensus.cc:374] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:52.853751 15389 raft_consensus.cc:729] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fcceeb71cf4f6c9cf5cef9ba1e3d3a, State: Initialized, Role: FOLLOWER
I20260806 16:03:52.853881 15392 raft_consensus.cc:348] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:52.854130 15392 raft_consensus.cc:374] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:52.854189 15392 raft_consensus.cc:729] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43f14b585e2433882dc3b1ca6a08699, State: Initialized, Role: FOLLOWER
I20260806 16:03:52.854223 15389 consensus_queue.cc:260] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:52.854661 15392 consensus_queue.cc:260] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:52.855041 14965 heartbeater.cc:503] Master 127.13.148.62:40809 was elected leader, sending a full tablet report...
I20260806 16:03:52.855314 15389 ts_tablet_manager.cc:1428] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:03:52.856209 15392 ts_tablet_manager.cc:1428] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:03:52.856258 15165 heartbeater.cc:503] Master 127.13.148.62:40809 was elected leader, sending a full tablet report...
I20260806 16:03:52.856410 15390 raft_consensus.cc:348] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:52.856671 15390 raft_consensus.cc:374] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:52.856724 15390 raft_consensus.cc:729] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81921810119e481a9c2352fe329433dc, State: Initialized, Role: FOLLOWER
I20260806 16:03:52.857103 15390 consensus_queue.cc:260] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:52.857793 15366 heartbeater.cc:503] Master 127.13.148.62:40809 was elected leader, sending a full tablet report...
I20260806 16:03:52.858021 15390 ts_tablet_manager.cc:1428] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260806 16:03:53.011492 15166 tablet.cc:2369] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:03:53.014801 15368 tablet.cc:2369] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:03:53.014881 14966 tablet.cc:2369] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:03:53.017978 15399 raft_consensus.cc:482] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:03:53.018321 15399 raft_consensus.cc:504] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:53.019528 15399 leader_election.cc:288] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40fcceeb71cf4f6c9cf5cef9ba1e3d3a (127.13.148.1:36037), b43f14b585e2433882dc3b1ca6a08699 (127.13.148.2:37645)
I20260806 16:03:53.024901 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:53.019837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:36037 (local address 127.13.148.3:56737)
0806 16:03:53.019977 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:53.019998 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:53.020103 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:53.020447 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:53.020453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:53.020479 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:53.020628 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:53.020637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.021718 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.021721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:53.023329 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:53.023342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.024314 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.024318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:53.024477 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:53.024522 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 16:03:53.024617 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 16:03:53.024706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":114,"spinlock_wait_cycles":4480}
I20260806 16:03:53.024910 15400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:53.019837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37645 (local address 127.13.148.3:40397)
0806 16:03:53.020407 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:53.020421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:53.020523 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:53.020746 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:53.020751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:53.020771 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:53.020871 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:53.020875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.021914 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.021917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:53.023329 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:53.023342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.024314 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.024318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:53.024480 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:53.024526 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 16:03:53.024658 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 16:03:53.024708 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":226,"thread_start_us":188,"threads_started":1}
I20260806 16:03:53.024964 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:53.019956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:40397 (local address 127.13.148.2:37645)
0806 16:03:53.020099 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:53.020104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:53.020528 (+ 424us) server_negotiation.cc:407] Connection header received
0806 16:03:53.020601 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:53.020603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:53.020645 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:53.020703 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:53.020985 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.021715 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.023494 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.024125 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.024267 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:53.024323 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:53.024729 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 16:03:53.024821 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 16:03:53.024964 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:53.019949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:56737 (local address 127.13.148.1:36037)
0806 16:03:53.020085 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:53.020091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:53.020111 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:03:53.020263 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:53.020266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:53.020331 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:53.020398 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:53.020782 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.021578 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.023494 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.024167 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.024276 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:53.024323 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:53.024729 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 16:03:53.024821 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:03:53.025870 14907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee36e885f3846b6a47449e05a7a8740" candidate_uuid: "81921810119e481a9c2352fe329433dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" is_pre_election: true
I20260806 16:03:53.025870 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee36e885f3846b6a47449e05a7a8740" candidate_uuid: "81921810119e481a9c2352fe329433dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43f14b585e2433882dc3b1ca6a08699" is_pre_election: true
I20260806 16:03:53.026288 15087 raft_consensus.cc:2454] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81921810119e481a9c2352fe329433dc in term 0.
I20260806 16:03:53.026299 14907 raft_consensus.cc:2454] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81921810119e481a9c2352fe329433dc in term 0.
I20260806 16:03:53.027007 15204 leader_election.cc:302] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81921810119e481a9c2352fe329433dc, b43f14b585e2433882dc3b1ca6a08699; no voters:
I20260806 16:03:53.027555 15399 raft_consensus.cc:2781] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:03:53.027632 15399 raft_consensus.cc:482] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:03:53.027663 15399 raft_consensus.cc:3037] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:53.028793 15399 raft_consensus.cc:504] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:53.029213 15399 leader_election.cc:288] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [CANDIDATE]: Term 1 election: Requested vote from peers 40fcceeb71cf4f6c9cf5cef9ba1e3d3a (127.13.148.1:36037), b43f14b585e2433882dc3b1ca6a08699 (127.13.148.2:37645)
I20260806 16:03:53.029597 15087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee36e885f3846b6a47449e05a7a8740" candidate_uuid: "81921810119e481a9c2352fe329433dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43f14b585e2433882dc3b1ca6a08699"
I20260806 16:03:53.029665 14907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ee36e885f3846b6a47449e05a7a8740" candidate_uuid: "81921810119e481a9c2352fe329433dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a"
I20260806 16:03:53.029753 15087 raft_consensus.cc:3037] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:53.029825 14907 raft_consensus.cc:3037] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:53.030941 15087 raft_consensus.cc:2454] T 2ee36e885f3846b6a47449e05a7a8740 P b43f14b585e2433882dc3b1ca6a08699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81921810119e481a9c2352fe329433dc in term 1.
I20260806 16:03:53.031095 14907 raft_consensus.cc:2454] T 2ee36e885f3846b6a47449e05a7a8740 P 40fcceeb71cf4f6c9cf5cef9ba1e3d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81921810119e481a9c2352fe329433dc in term 1.
I20260806 16:03:53.031435 15204 leader_election.cc:302] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 40fcceeb71cf4f6c9cf5cef9ba1e3d3a, 81921810119e481a9c2352fe329433dc, b43f14b585e2433882dc3b1ca6a08699; no voters:
I20260806 16:03:53.031629 15399 raft_consensus.cc:2781] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:53.031930 15399 raft_consensus.cc:686] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [term 1 LEADER]: Becoming Leader. State: Replica: 81921810119e481a9c2352fe329433dc, State: Running, Role: LEADER
I20260806 16:03:53.032320 15399 consensus_queue.cc:237] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } }
I20260806 16:03:53.035992 14695 catalog_manager.cc:5495] T 2ee36e885f3846b6a47449e05a7a8740 P 81921810119e481a9c2352fe329433dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 81921810119e481a9c2352fe329433dc (127.13.148.3). New cstate: current_term: 1 leader_uuid: "81921810119e481a9c2352fe329433dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40fcceeb71cf4f6c9cf5cef9ba1e3d3a" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43f14b585e2433882dc3b1ca6a08699" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81921810119e481a9c2352fe329433dc" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37263 } health_report { overall_health: HEALTHY } } }
Aug 06 16:03:53 dist-test-slave-7kzw krb5kdc[14630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:03:53.074842 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:53.064075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.13.148.1:36037)
0806 16:03:53.064296 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:53.064302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:53.064467 (+ 165us) server_negotiation.cc:407] Connection header received
0806 16:03:53.064716 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:53.064719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:53.064768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:53.064868 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:53.065685 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.066261 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.067314 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.067588 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.067722 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:53.070630 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:53.070644 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:53.070661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:53.070690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:53.072514 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:53.073122 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:53.073129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:53.073132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:53.073249 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:53.073674 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:53.073678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:53.073682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:53.073972 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:53.074240 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:53.074507 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 16:03:53.074666 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
Aug 06 16:03:53 dist-test-slave-7kzw krb5kdc[14630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:03:53 dist-test-slave-7kzw krb5kdc[14630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:03:53.108063 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:53.098573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.13.148.1:36037)
0806 16:03:53.098777 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:53.098784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:53.098860 (+ 76us) server_negotiation.cc:407] Connection header received
0806 16:03:53.098998 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:53.099000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:53.099049 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:53.099164 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:53.099859 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.100453 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.101345 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.101594 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.101704 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:53.104551 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:53.104561 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:53.104563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:53.104594 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:53.106058 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:53.106628 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:53.106635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:53.106637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:53.106739 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:53.107046 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:53.107049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:53.107050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:53.107235 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:53.107362 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:53.107809 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 16:03:53.107929 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 16:03:53 dist-test-slave-7kzw krb5kdc[14630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:03:53 dist-test-slave-7kzw krb5kdc[14630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:03:53.140621 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:53.130969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.13.148.1:36037)
0806 16:03:53.131154 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:53.131159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:53.131337 (+ 178us) server_negotiation.cc:407] Connection header received
0806 16:03:53.131529 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:53.131531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:53.131578 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:53.131710 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:53.132445 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.132958 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.133821 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:53.134044 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:53.134163 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:53.136985 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:53.136988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:53.136990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:53.137018 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:53.138680 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:53.139207 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:53.139209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:53.139210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:53.139287 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:53.139635 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:53.139639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:53.139640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:53.139849 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:53.139969 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:53.140316 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 16:03:53.140440 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 16:03:53.142421 14854 tablet_service.cc:2908] Rejecting scan request for tablet 2ee36e885f3846b6a47449e05a7a8740: Uninitialized: clean time has not yet been initialized
I20260806 16:03:53.145587 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 14787
I20260806 16:03:53.160851 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 14970
I20260806 16:03:53.171196 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 15171
I20260806 16:03:53.185290 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 14647
2026-08-06T16:03:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2882 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:03:53 dist-test-slave-7kzw krb5kdc[15421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:03:53 dist-test-slave-7kzw krb5kdc[15421](info): set up 2 sockets
Aug 06 16:03:53 dist-test-slave-7kzw krb5kdc[15421](info): commencing operation
krb5kdc: starting...
W20260806 16:03:54.452420 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:03:54 dist-test-slave-7kzw krb5kdc[15421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:03:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:03:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:03:54.590093 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35543
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:54.714358 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:54.714735 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:54.714780 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:54.719048 15438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:03:54.719111 15438 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 16:03:54.719129 15438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:54.719149 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:54.719205 15438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:03:54.719234 15438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:03:54.724762 15438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35543
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:54.725867 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:54.726989 15438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:54.736058 15450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:54.736128 15438 server_base.cc:1034] running on GCE node
W20260806 16:03:54.736073 15445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:54.736109 15446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:54.737111 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:54.738407 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:54.739660 15438 hybrid_clock.cc:648] HybridClock initialized: now 1786032234739609 us; error 66 us; skew 500 ppm
Aug 06 16:03:54 dist-test-slave-7kzw krb5kdc[15421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032234, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:54.742933 15438 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:03:54.743927 15438 webserver.cc:460] Webserver started at http://127.13.148.62:43427/ using document root <none> and password file <none>
I20260806 16:03:54.744570 15438 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:54.744678 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:54.744962 15438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:54.747385 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "24321c420d264e318431340b2cbcf697"
format_stamp: "Formatted at 2026-08-06 16:03:54 on dist-test-slave-7kzw"
server_key: "51e557278d7b22a0b9fb126ff3189de3"
server_key_iv: "46d18216c8f65423a4482cc9b526486d"
server_key_version: "encryptionkey@0"
I20260806 16:03:54.748056 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "24321c420d264e318431340b2cbcf697"
format_stamp: "Formatted at 2026-08-06 16:03:54 on dist-test-slave-7kzw"
server_key: "51e557278d7b22a0b9fb126ff3189de3"
server_key_iv: "46d18216c8f65423a4482cc9b526486d"
server_key_version: "encryptionkey@0"
I20260806 16:03:54.752336 15438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:03:54.755193 15459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:54.756515 15438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:54.756711 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "24321c420d264e318431340b2cbcf697"
format_stamp: "Formatted at 2026-08-06 16:03:54 on dist-test-slave-7kzw"
server_key: "51e557278d7b22a0b9fb126ff3189de3"
server_key_iv: "46d18216c8f65423a4482cc9b526486d"
server_key_version: "encryptionkey@0"
I20260806 16:03:54.756844 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:54.769217 15438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:54.772559 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:54.772799 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:54.781589 15438 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35543
I20260806 16:03:54.781649 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35543 every 8 connection(s)
I20260806 16:03:54.782692 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:03:54.785701 15569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:54.787724 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 15438
I20260806 16:03:54.787875 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:03:54.788172 13904 external_mini_cluster.cc:1428] Setting key 7bcf7d0da751088a93d13845d932b7c9
I20260806 16:03:54.792302 15569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697: Bootstrap starting.
I20260806 16:03:54.795012 15569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:54.795982 15569 log.cc:824] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:03:54 dist-test-slave-7kzw krb5kdc[15421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:54.798406 15569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697: No bootstrap required, opened a new log
I20260806 16:03:54.802052 15569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24321c420d264e318431340b2cbcf697" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35543 } }
I20260806 16:03:54.802338 15569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:54.802399 15569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24321c420d264e318431340b2cbcf697, State: Initialized, Role: FOLLOWER
I20260806 16:03:54.802958 15569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24321c420d264e318431340b2cbcf697" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35543 } }
I20260806 16:03:54.803134 15569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:03:54.803236 15569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:03:54.803361 15569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:54.803856 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:54.789769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.13.148.62:35543)
0806 16:03:54.790211 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:54.790240 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:54.790278 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:03:54.791202 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:54.791222 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:54.791563 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:54.791858 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:54.792853 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:54.793966 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:54.795240 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:54.795739 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:54.795879 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:54.798731 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:54.798737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:54.798751 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:54.798779 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:54.800611 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:54.801118 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:54.801124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:54.801128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:54.801280 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:54.801593 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:54.801601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:54.801602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:54.802012 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:54.802252 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:54.802571 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 16:03:54.802874 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":148,"threads_started":1}
I20260806 16:03:54.804422 15569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24321c420d264e318431340b2cbcf697" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35543 } }
I20260806 16:03:54.804771 15569 leader_election.cc:302] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24321c420d264e318431340b2cbcf697; no voters:
I20260806 16:03:54.805200 15569 leader_election.cc:288] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:03:54.805320 15576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:54.805557 15576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [term 1 LEADER]: Becoming Leader. State: Replica: 24321c420d264e318431340b2cbcf697, State: Running, Role: LEADER
I20260806 16:03:54.805900 15576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24321c420d264e318431340b2cbcf697" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35543 } }
I20260806 16:03:54.806541 15569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:03:54.807328 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24321c420d264e318431340b2cbcf697. Latest consensus state: current_term: 1 leader_uuid: "24321c420d264e318431340b2cbcf697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24321c420d264e318431340b2cbcf697" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35543 } } }
I20260806 16:03:54.807468 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [sys.catalog]: This master's current role is: LEADER
I20260806 16:03:54.807729 15577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24321c420d264e318431340b2cbcf697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24321c420d264e318431340b2cbcf697" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35543 } } }
I20260806 16:03:54.807843 15577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697 [sys.catalog]: This master's current role is: LEADER
W20260806 16:03:54.811434 15594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:03:54.811520 15594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:03:54.811605 15586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:03:54.812333 15586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:03:54.817821 15586 catalog_manager.cc:1347] Generated new cluster ID: 4f917ea1843c4bae9fb104134462290a
I20260806 16:03:54.817900 15586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:03:54.837038 15586 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:03:54.838037 15586 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:03:54.842639 15586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697: Generated new TSK 0
I20260806 16:03:54.843521 15586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:03:54.888200 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35543
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:55.006613 15602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:55.006922 15602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:55.006970 15602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:55.011993 15602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:55.012071 15602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:55.012115 15602 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:03:55.012149 15602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:03:55.017617 15602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:55.018846 15602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:55.019954 15602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:55.030531 15616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:55.030642 15602 server_base.cc:1034] running on GCE node
W20260806 16:03:55.030537 15618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:55.030534 15613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:55.031342 15602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:55.031987 15602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:55.033182 15602 hybrid_clock.cc:648] HybridClock initialized: now 1786032235033159 us; error 38 us; skew 500 ppm
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:55.036329 15602 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:03:55.037230 15602 webserver.cc:460] Webserver started at http://127.13.148.1:44487/ using document root <none> and password file <none>
I20260806 16:03:55.037827 15602 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:55.037895 15602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:55.038117 15602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:55.040225 15602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "a7fe39c336ce48c48672666a3970fcaa"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "17a8e13f38c56d34d011e764fc9bcc61"
server_key_iv: "2426839bf4ecddf358bbea9e13ada471"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.040805 15602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "a7fe39c336ce48c48672666a3970fcaa"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "17a8e13f38c56d34d011e764fc9bcc61"
server_key_iv: "2426839bf4ecddf358bbea9e13ada471"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.044653 15602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:03:55.047240 15627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:55.048609 15602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:55.048748 15602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "a7fe39c336ce48c48672666a3970fcaa"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "17a8e13f38c56d34d011e764fc9bcc61"
server_key_iv: "2426839bf4ecddf358bbea9e13ada471"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.048861 15602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:55.067308 15602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:55.070917 15602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:55.071151 15602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:55.072705 15602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:55.072789 15602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:55.072871 15602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:55.072916 15602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:55.083889 15602 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:46669
I20260806 16:03:55.084003 15794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:46669 every 8 connection(s)
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:55.085065 15602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:03:55.086885 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 15602
I20260806 16:03:55.087002 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:03:55.087352 13904 external_mini_cluster.cc:1428] Setting key 3d82cb1512ef471efa3bcd4ed6b1e64b
I20260806 16:03:55.089879 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.073835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:32875 (local address 127.13.148.62:35543)
0806 16:03:55.073983 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.073987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.075163 (+ 1176us) server_negotiation.cc:407] Connection header received
0806 16:03:55.076620 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.076626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.076700 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.076795 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:55.078950 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.079906 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.081568 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.082007 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.082135 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.085441 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:55.085446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:55.085449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:55.085487 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:55.087357 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.088015 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.088019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.088020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.088119 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.088512 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.088515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.088516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.088771 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:55.088865 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.089455 (+ 590us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.089622 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:03:55.090572 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.074331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35543 (local address 127.13.148.1:32875)
0806 16:03:55.074898 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.074946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.076302 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.077240 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.077252 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.077838 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:55.078683 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.078707 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.080116 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.080121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.081396 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.081406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.082221 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.082228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.082468 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.083200 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:55.083217 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:55.085247 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:55.087506 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.087511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.087523 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.087894 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.088221 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.088224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.088226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.088411 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.088886 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:55.088893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:55.089180 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.089344 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.089607 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":147,"threads_started":1}
I20260806 16:03:55.092132 15795 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35543
I20260806 16:03:55.092300 15795 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:55.092895 15795 heartbeater.cc:511] Master 127.13.148.62:35543 requested a full tablet report, sending...
I20260806 16:03:55.094569 15492 ts_manager.cc:194] Registered new tserver with Master: a7fe39c336ce48c48672666a3970fcaa (127.13.148.1:46669)
I20260806 16:03:55.095952 15492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:32875
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:03:55.103909 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.094700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35543 (local address 127.13.148.1:37307)
0806 16:03:55.094995 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.095011 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.095126 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.095403 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.095406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.095638 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:55.095804 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.095816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.096794 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.096801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.097836 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.097851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.098415 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.098418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.098561 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.099123 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:55.099143 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:55.099631 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:55.101882 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.101884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.101886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.102136 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.102640 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.102643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.102644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.102759 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.103326 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:55.103332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:55.103459 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.103591 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.103748 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260806 16:03:55.104070 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.094789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:37307 (local address 127.13.148.62:35543)
0806 16:03:55.094912 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.094918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.095078 (+ 160us) server_negotiation.cc:407] Connection header received
0806 16:03:55.095228 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.095230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.095274 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.095357 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:55.095941 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.096647 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.097991 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.098284 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.098366 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.099801 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:55.099805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:55.099807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:55.099831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:55.101757 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.102259 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.102262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.102263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.102514 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.102963 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.102969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.102973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.103210 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:55.103319 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.103700 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.103839 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:03:55.122857 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35543
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:55.245206 15809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:55.245556 15809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:55.245605 15809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:55.249682 15809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:55.249756 15809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:55.249797 15809 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:03:55.249831 15809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:03:55.256210 15809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:55.257441 15809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:55.258939 15809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:55.269827 15820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:55.269833 15822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:55.270077 15809 server_base.cc:1034] running on GCE node
W20260806 16:03:55.269833 15819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:55.270823 15809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:55.271502 15809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:55.272717 15809 hybrid_clock.cc:648] HybridClock initialized: now 1786032235272695 us; error 33 us; skew 500 ppm
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:55.275686 15809 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:03:55.276531 15809 webserver.cc:460] Webserver started at http://127.13.148.2:44047/ using document root <none> and password file <none>
I20260806 16:03:55.277081 15809 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:55.277145 15809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:55.277312 15809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:55.279507 15809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "b31ae33e599b4e438fb48e8843bb0aad"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "b4272c332e5eee7ffde45674f9fb7bdf"
server_key_iv: "2b588dd38157cee3842165645356291e"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.280067 15809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "b31ae33e599b4e438fb48e8843bb0aad"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "b4272c332e5eee7ffde45674f9fb7bdf"
server_key_iv: "2b588dd38157cee3842165645356291e"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.284121 15809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 16:03:55.286856 15835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:55.288306 15809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:55.288435 15809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "b31ae33e599b4e438fb48e8843bb0aad"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "b4272c332e5eee7ffde45674f9fb7bdf"
server_key_iv: "2b588dd38157cee3842165645356291e"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.288545 15809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:55.309096 15809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:55.312911 15809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:55.313103 15809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:55.314859 15809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:55.314957 15809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:55.315037 15809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:55.315085 15809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:55.327874 15809 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:45259
I20260806 16:03:55.328342 16017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:45259 every 8 connection(s)
I20260806 16:03:55.328986 15809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:03:55.331650 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 15809
I20260806 16:03:55.331684 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.315842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:38527 (local address 127.13.148.62:35543)
0806 16:03:55.315991 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.315995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.317188 (+ 1193us) server_negotiation.cc:407] Connection header received
0806 16:03:55.318275 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.318278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.318341 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.318434 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:55.320482 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.321201 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.323466 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.323840 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.323996 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.327139 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:55.327143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:55.327146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:55.327189 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:55.329124 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.329827 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.329831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.329832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.329940 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.330366 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.330368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.330370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.330665 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:55.330753 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.331426 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.331549 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:03:55.331776 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:03:55.332055 13904 external_mini_cluster.cc:1428] Setting key 9e0d06190474c455d7ce7c5ed3d151f5
I20260806 16:03:55.332847 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.316454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35543 (local address 127.13.148.2:38527)
0806 16:03:55.316992 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.317038 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.318041 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.318727 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.318741 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.319416 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:55.320246 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.320275 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.321366 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.321371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.323278 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.323292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.324063 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.324067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.324301 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.325071 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:55.325091 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:55.326914 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:55.329268 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.329275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.329291 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.329684 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.330068 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.330071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.330073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.330248 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.330859 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:55.330864 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:55.331118 (+ 254us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.331300 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.331720 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":141,"threads_started":1}
I20260806 16:03:55.334403 16018 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35543
I20260806 16:03:55.334613 16018 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:55.335265 16018 heartbeater.cc:511] Master 127.13.148.62:35543 requested a full tablet report, sending...
I20260806 16:03:55.336490 15492 ts_manager.cc:194] Registered new tserver with Master: b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259)
I20260806 16:03:55.337263 15492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:38527
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:03:55.346068 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.336929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:56631 (local address 127.13.148.62:35543)
0806 16:03:55.337631 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.337636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.337886 (+ 250us) server_negotiation.cc:407] Connection header received
0806 16:03:55.338035 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.338041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.338104 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.338199 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:55.338802 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.339677 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.340569 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.340805 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.340892 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.342395 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:55.342401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:55.342404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:55.342436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:55.344162 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.344690 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.344693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.344695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.344810 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.345115 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.345120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.345122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.345359 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:55.345476 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.345780 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.345897 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621}
I20260806 16:03:55.346081 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.336819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35543 (local address 127.13.148.2:56631)
0806 16:03:55.337795 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.337808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.337934 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.338255 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.338261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.338537 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:55.338667 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.338675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.339798 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.339801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.340437 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.340452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.340939 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.340946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.341079 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.341654 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:55.341674 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:55.342185 (+ 511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:55.344306 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.344312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.344315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.344531 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.344917 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.344921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.344923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.345018 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.345464 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:55.345467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:55.345573 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.345718 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.345885 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:03:55.367326 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35543
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:55.487907 16031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:55.488231 16031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:55.488284 16031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:55.493335 16031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:55.493454 16031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:55.493530 16031 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:03:55.493607 16031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:03:55.499765 16031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:55.501009 16031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:55.502195 16031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:55.513417 16046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:55.513746 16031 server_base.cc:1034] running on GCE node
W20260806 16:03:55.513435 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:55.513435 16039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:55.515461 16031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:55.516193 16031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:55.517382 16031 hybrid_clock.cc:648] HybridClock initialized: now 1786032235517350 us; error 58 us; skew 500 ppm
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:55.520749 16031 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:03:55.521809 16031 webserver.cc:460] Webserver started at http://127.13.148.3:44307/ using document root <none> and password file <none>
I20260806 16:03:55.522549 16031 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:55.522630 16031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:55.522884 16031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:55.525247 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "8020209fcc454974a566bd40e399e1d9"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "75c4c70c3f52a3bcae557743989645f9"
server_key_iv: "afdaf1d5c7c6763b5c65a43bb07c1ca7"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.525938 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "8020209fcc454974a566bd40e399e1d9"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "75c4c70c3f52a3bcae557743989645f9"
server_key_iv: "afdaf1d5c7c6763b5c65a43bb07c1ca7"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.530381 16031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 16:03:55.533264 16055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:55.534723 16031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 16:03:55.534878 16031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "8020209fcc454974a566bd40e399e1d9"
format_stamp: "Formatted at 2026-08-06 16:03:55 on dist-test-slave-7kzw"
server_key: "75c4c70c3f52a3bcae557743989645f9"
server_key_iv: "afdaf1d5c7c6763b5c65a43bb07c1ca7"
server_key_version: "encryptionkey@0"
I20260806 16:03:55.535007 16031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:55.554304 16031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:55.557677 16031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:55.557912 16031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:55.560195 16031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:55.560271 16031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:55.560348 16031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:55.560387 16031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:55.573760 16031 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:45969
I20260806 16:03:55.573784 16233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:45969 every 8 connection(s)
I20260806 16:03:55.574832 16031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:03:55.575465 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.561302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:54233 (local address 127.13.148.62:35543)
0806 16:03:55.561451 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.561455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.562658 (+ 1203us) server_negotiation.cc:407] Connection header received
0806 16:03:55.563757 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.563778 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.563853 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.563911 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:55.565765 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.566579 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.567730 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.568031 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.568151 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.571511 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:55.571515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:55.571517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:55.571547 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:55.573116 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.573657 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.573660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.573664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.573755 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.574138 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.574145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.574148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.574347 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:55.574457 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.575032 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.575207 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:03:55.576027 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 16031
I20260806 16:03:55.576162 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:03:55.576445 13904 external_mini_cluster.cc:1428] Setting key 5feeed2615788996847f5d69b2bc6fd3
I20260806 16:03:55.576242 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.561945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35543 (local address 127.13.148.3:54233)
0806 16:03:55.562504 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.562562 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.563451 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.564394 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.564401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.564959 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:55.565610 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.565642 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.566705 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.566709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.567593 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.567602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.568227 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.568237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.568499 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.569249 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:55.569278 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:55.571322 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:55.573261 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.573267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.573282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.573546 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.573859 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.573862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.573864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.574037 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.574456 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:55.574490 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:55.574751 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.574949 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.575234 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":147,"threads_started":1}
I20260806 16:03:55.578191 16234 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35543
I20260806 16:03:55.578382 16234 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:55.579030 16234 heartbeater.cc:511] Master 127.13.148.62:35543 requested a full tablet report, sending...
I20260806 16:03:55.580276 15490 ts_manager.cc:194] Registered new tserver with Master: 8020209fcc454974a566bd40e399e1d9 (127.13.148.3:45969)
I20260806 16:03:55.581035 15490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:54233
I20260806 16:03:55.588500 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.580711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35543 (local address 127.13.148.3:37905)
0806 16:03:55.581087 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.581104 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.581256 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.581584 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.581589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.581812 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:55.581926 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.581932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.582824 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.582828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.583440 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.583449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.583965 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.583969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.584072 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.584678 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:55.584691 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:55.585046 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:55.586997 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.587000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.587002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.587230 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.587535 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.587538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.587539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.587626 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.588044 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:55.588046 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:55.588130 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.588230 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.588351 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
I20260806 16:03:55.588574 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.580789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:37905 (local address 127.13.148.62:35543)
0806 16:03:55.580917 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.580921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.581217 (+ 296us) server_negotiation.cc:407] Connection header received
0806 16:03:55.581370 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.581373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.581421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.581523 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:55.582062 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.582692 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.583595 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.583842 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.583961 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.585215 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:55.585220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:55.585223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:55.585250 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:55.586877 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.587349 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.587353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.587354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.587443 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.587760 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.587763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.587764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.587950 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:55.588039 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.588322 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.588432 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:03:55.591156 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:55.622184 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.613342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.13.148.62:35543)
0806 16:03:55.613496 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.613500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.613515 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:03:55.613567 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.613570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.613615 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.613695 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:55.614590 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.615176 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.616069 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.616286 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.616373 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.618970 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:55.618973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:55.618976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:55.619000 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:55.620508 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.620912 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.620915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.620916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.620992 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.621244 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.621246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.621247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.621412 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:55.621515 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.621853 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.621949 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 16:03:55.625523 15490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:03:55.628994 15490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:03:55.645197 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.638775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45259 (local address 127.0.0.1:40624)
0806 16:03:55.639560 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.639615 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.639750 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.640668 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.640674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.640704 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:55.640835 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.640849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.642175 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.642179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.643733 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.643745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.644635 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.644639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.644746 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.644784 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.644899 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.645021 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260806 16:03:55.645241 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.638940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45969 (local address 127.0.0.1:35822)
0806 16:03:55.640078 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.640093 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.640173 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.641100 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.641103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.641116 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:55.641213 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.641219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.642544 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.642548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.643706 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.643716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.644905 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.644909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.644986 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.645005 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.645078 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.645132 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":47,"thread_start_us":163,"threads_started":1}
I20260806 16:03:55.645578 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.638882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.13.148.2:45259)
0806 16:03:55.639659 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.639673 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.639710 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:03:55.640257 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.640264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.640480 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.640657 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:55.640957 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.642044 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.643880 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.644504 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.644637 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.644822 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.645319 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.645421 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":128,"threads_started":1}
I20260806 16:03:55.645509 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.639629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.13.148.3:45969)
0806 16:03:55.640180 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.640192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.640219 (+ 27us) server_negotiation.cc:407] Connection header received
0806 16:03:55.640300 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.640306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.640502 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.640660 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:55.641339 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.642363 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.643839 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.644626 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.644748 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.644929 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.645129 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.645244 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":184,"threads_started":1}
I20260806 16:03:55.645989 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.639639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.13.148.1:46669)
0806 16:03:55.640129 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.640136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.640987 (+ 851us) server_negotiation.cc:407] Connection header received
0806 16:03:55.641109 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.641114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.641309 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.641467 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:55.641681 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.642597 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.644006 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.644748 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.644844 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.644959 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.645551 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.645837 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":175,"threads_started":1}
I20260806 16:03:55.646271 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.639214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46669 (local address 127.0.0.1:51344)
0806 16:03:55.640859 (+ 1645us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.640871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.640998 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.641444 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.641447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.641466 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:55.641564 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.641572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.642774 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.642777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.643895 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.643903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.645026 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.645029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.645134 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.645159 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.645240 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.645306 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260806 16:03:55.648603 15702 tablet_service.cc:1467] Processing CreateTablet for tablet e9ac2b5906c84290a4def51ccb867c34 (DEFAULT_TABLE table=test-table [id=b54a4eee9c4b4f20a54c76828dcdc654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:55.648638 15917 tablet_service.cc:1467] Processing CreateTablet for tablet e9ac2b5906c84290a4def51ccb867c34 (DEFAULT_TABLE table=test-table [id=b54a4eee9c4b4f20a54c76828dcdc654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:55.648638 16134 tablet_service.cc:1467] Processing CreateTablet for tablet e9ac2b5906c84290a4def51ccb867c34 (DEFAULT_TABLE table=test-table [id=b54a4eee9c4b4f20a54c76828dcdc654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:55.649605 15702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ac2b5906c84290a4def51ccb867c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.649632 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ac2b5906c84290a4def51ccb867c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.649639 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ac2b5906c84290a4def51ccb867c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.656313 16265 tablet_bootstrap.cc:492] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: Bootstrap starting.
I20260806 16:03:55.658193 16269 tablet_bootstrap.cc:492] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: Bootstrap starting.
I20260806 16:03:55.658761 16265 tablet_bootstrap.cc:654] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.658802 16266 tablet_bootstrap.cc:492] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: Bootstrap starting.
I20260806 16:03:55.659503 16265 log.cc:824] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:55.660491 16269 tablet_bootstrap.cc:654] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.660872 16266 tablet_bootstrap.cc:654] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.661816 16266 log.cc:824] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:55.661943 16265 tablet_bootstrap.cc:492] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: No bootstrap required, opened a new log
I20260806 16:03:55.662137 16265 ts_tablet_manager.cc:1397] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:03:55.663997 16266 tablet_bootstrap.cc:492] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: No bootstrap required, opened a new log
I20260806 16:03:55.664179 16266 ts_tablet_manager.cc:1397] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:03:55.665714 16269 log.cc:824] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:55.666970 16265 raft_consensus.cc:348] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.667238 16265 raft_consensus.cc:374] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.667317 16265 raft_consensus.cc:729] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fe39c336ce48c48672666a3970fcaa, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.667765 16265 consensus_queue.cc:260] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.668687 16269 tablet_bootstrap.cc:492] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: No bootstrap required, opened a new log
I20260806 16:03:55.668815 16265 ts_tablet_manager.cc:1428] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 16:03:55.668870 16269 ts_tablet_manager.cc:1397] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260806 16:03:55.668859 15795 heartbeater.cc:503] Master 127.13.148.62:35543 was elected leader, sending a full tablet report...
I20260806 16:03:55.669303 16266 raft_consensus.cc:348] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.669534 16266 raft_consensus.cc:374] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.669595 16266 raft_consensus.cc:729] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31ae33e599b4e438fb48e8843bb0aad, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.670100 16266 consensus_queue.cc:260] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.670814 16018 heartbeater.cc:503] Master 127.13.148.62:35543 was elected leader, sending a full tablet report...
I20260806 16:03:55.671080 16266 ts_tablet_manager.cc:1428] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:03:55.673264 16269 raft_consensus.cc:348] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.673516 16269 raft_consensus.cc:374] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.673575 16269 raft_consensus.cc:729] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8020209fcc454974a566bd40e399e1d9, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.673970 16269 consensus_queue.cc:260] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.674650 16234 heartbeater.cc:503] Master 127.13.148.62:35543 was elected leader, sending a full tablet report...
I20260806 16:03:55.674894 16269 ts_tablet_manager.cc:1428] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:03:55.769724 16277 raft_consensus.cc:482] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:03:55.769951 16277 raft_consensus.cc:504] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.771113 16277 leader_election.cc:288] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259), a7fe39c336ce48c48672666a3970fcaa (127.13.148.1:46669)
I20260806 16:03:55.775321 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.771454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45259 (local address 127.13.148.3:34333)
0806 16:03:55.771633 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.771649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.771773 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.772110 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.772113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.772135 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:55.772267 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.772276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.773131 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.773135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.774097 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.774107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.774927 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.774931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.775046 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.775081 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.775150 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.775213 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":1}
I20260806 16:03:55.775398 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.771572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:34333 (local address 127.13.148.2:45259)
0806 16:03:55.771724 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.771730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.771751 (+ 21us) server_negotiation.cc:407] Connection header received
0806 16:03:55.771910 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.771922 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.771988 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.772077 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:55.772391 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.773006 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.774229 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.774807 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.774917 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.774959 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.775187 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.775269 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:03:55.776041 15950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ac2b5906c84290a4def51ccb867c34" candidate_uuid: "8020209fcc454974a566bd40e399e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ae33e599b4e438fb48e8843bb0aad" is_pre_election: true
I20260806 16:03:55.776295 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.771701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:51677 (local address 127.13.148.1:46669)
0806 16:03:55.771864 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.771870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.772288 (+ 418us) server_negotiation.cc:407] Connection header received
0806 16:03:55.772393 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.772396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.772448 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.772531 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:55.773310 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.773775 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.775117 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.775741 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.775867 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.775914 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.776103 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.776186 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:03:55.776291 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.771454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46669 (local address 127.13.148.3:51677)
0806 16:03:55.772183 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.772196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.772293 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.772545 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.772547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.772560 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:55.773133 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.773140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.773890 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.773894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.774985 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.774994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.775866 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.775870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.775978 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.775995 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.776086 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.776150 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":333,"thread_start_us":194,"threads_started":1}
I20260806 16:03:55.776399 15950 raft_consensus.cc:2454] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8020209fcc454974a566bd40e399e1d9 in term 0.
I20260806 16:03:55.776957 16060 leader_election.cc:302] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8020209fcc454974a566bd40e399e1d9, b31ae33e599b4e438fb48e8843bb0aad; no voters:
I20260806 16:03:55.776952 15729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ac2b5906c84290a4def51ccb867c34" candidate_uuid: "8020209fcc454974a566bd40e399e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fe39c336ce48c48672666a3970fcaa" is_pre_election: true
I20260806 16:03:55.777263 15729 raft_consensus.cc:2454] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8020209fcc454974a566bd40e399e1d9 in term 0.
I20260806 16:03:55.777246 16277 raft_consensus.cc:2781] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:03:55.777351 16277 raft_consensus.cc:482] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:03:55.777396 16277 raft_consensus.cc:3037] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:55.778589 16277 raft_consensus.cc:504] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.778890 16277 leader_election.cc:288] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [CANDIDATE]: Term 1 election: Requested vote from peers b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259), a7fe39c336ce48c48672666a3970fcaa (127.13.148.1:46669)
I20260806 16:03:55.779279 15950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ac2b5906c84290a4def51ccb867c34" candidate_uuid: "8020209fcc454974a566bd40e399e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ae33e599b4e438fb48e8843bb0aad"
I20260806 16:03:55.779292 15729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ac2b5906c84290a4def51ccb867c34" candidate_uuid: "8020209fcc454974a566bd40e399e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fe39c336ce48c48672666a3970fcaa"
I20260806 16:03:55.779417 15729 raft_consensus.cc:3037] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:55.779417 15950 raft_consensus.cc:3037] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:55.780795 15729 raft_consensus.cc:2454] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8020209fcc454974a566bd40e399e1d9 in term 1.
I20260806 16:03:55.781208 15950 raft_consensus.cc:2454] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8020209fcc454974a566bd40e399e1d9 in term 1.
I20260806 16:03:55.781225 16058 leader_election.cc:302] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8020209fcc454974a566bd40e399e1d9, a7fe39c336ce48c48672666a3970fcaa; no voters:
I20260806 16:03:55.781461 16277 raft_consensus.cc:2781] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:55.781754 16277 raft_consensus.cc:686] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 8020209fcc454974a566bd40e399e1d9, State: Running, Role: LEADER
I20260806 16:03:55.782107 16277 consensus_queue.cc:237] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } }
I20260806 16:03:55.785355 15492 catalog_manager.cc:5495] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8020209fcc454974a566bd40e399e1d9 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } health_report { overall_health: UNKNOWN } } }
I20260806 16:03:55.805716 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.801442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.13.148.3:45969)
0806 16:03:55.801577 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.801580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.801596 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:03:55.802010 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.802013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.802081 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.802183 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:03:55.802616 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.803187 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.804115 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.804352 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.804445 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.804798 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:03:55.805237 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:03:55.805327 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.805493 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.805589 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:03:55.812623 15950 raft_consensus.cc:1261] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Refusing update from remote peer 8020209fcc454974a566bd40e399e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:55.812623 15729 raft_consensus.cc:1261] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Refusing update from remote peer 8020209fcc454974a566bd40e399e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:55.813577 16277 consensus_queue.cc:1035] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:55.813871 16279 consensus_queue.cc:1035] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:55.821272 16285 mvcc.cc:205] Tried to move back new op lower bound from 7315588037879820288 to 7315588037766782976. Current Snapshot: MvccSnapshot[applied={T|T < 7315588037879820288}]
I20260806 16:03:55.821839 16287 mvcc.cc:205] Tried to move back new op lower bound from 7315588037879820288 to 7315588037766782976. Current Snapshot: MvccSnapshot[applied={T|T < 7315588037879820288}]
I20260806 16:03:55.836344 15490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:03:55.836616 15490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:03:55.839694 15490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697: Sending DeleteTablet for 3 replicas of tablet e9ac2b5906c84290a4def51ccb867c34
I20260806 16:03:55.840596 15917 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ac2b5906c84290a4def51ccb867c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:55 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:40624
I20260806 16:03:55.840677 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ac2b5906c84290a4def51ccb867c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:55 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:35822
I20260806 16:03:55.840816 15702 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ac2b5906c84290a4def51ccb867c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:55 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51344
I20260806 16:03:55.841356 16305 tablet_replica.cc:330] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: stopping tablet replica
I20260806 16:03:55.841734 16305 raft_consensus.cc:2229] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:03:55.841902 15490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 16:03:55.842109 16305 raft_consensus.cc:2258] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 16:03:55.842365 15490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:03:55.843181 16306 tablet_replica.cc:330] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: stopping tablet replica
I20260806 16:03:55.843477 16306 raft_consensus.cc:2229] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:03:55.843871 16306 raft_consensus.cc:2258] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:03:55.843887 16305 ts_tablet_manager.cc:1905] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:55.847747 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 87aeb9871aa447a2b82a44ddfcd45416 (DEFAULT_TABLE table=test-table [id=b0ff086fa5e54173975bd900b3a74913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:55.847843 15917 tablet_service.cc:1467] Processing CreateTablet for tablet 87aeb9871aa447a2b82a44ddfcd45416 (DEFAULT_TABLE table=test-table [id=b0ff086fa5e54173975bd900b3a74913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:55.848058 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aeb9871aa447a2b82a44ddfcd45416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.848098 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aeb9871aa447a2b82a44ddfcd45416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.848196 15702 tablet_service.cc:1467] Processing CreateTablet for tablet 87aeb9871aa447a2b82a44ddfcd45416 (DEFAULT_TABLE table=test-table [id=b0ff086fa5e54173975bd900b3a74913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:55.848489 15702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87aeb9871aa447a2b82a44ddfcd45416. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.850780 16269 tablet_bootstrap.cc:492] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: Bootstrap starting.
I20260806 16:03:55.850940 16266 tablet_bootstrap.cc:492] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: Bootstrap starting.
I20260806 16:03:55.851775 16269 tablet_bootstrap.cc:654] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.851948 16266 tablet_bootstrap.cc:654] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.852227 16265 tablet_bootstrap.cc:492] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: Bootstrap starting.
I20260806 16:03:55.852793 16305 ts_tablet_manager.cc:1918] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:03:55.852917 16305 log.cc:1196] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/e9ac2b5906c84290a4def51ccb867c34
I20260806 16:03:55.853071 16269 tablet_bootstrap.cc:492] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: No bootstrap required, opened a new log
I20260806 16:03:55.853158 16269 ts_tablet_manager.cc:1397] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:03:55.853277 16305 ts_tablet_manager.cc:1939] T e9ac2b5906c84290a4def51ccb867c34 P b31ae33e599b4e438fb48e8843bb0aad: Deleting consensus metadata
I20260806 16:03:55.853396 16265 tablet_bootstrap.cc:654] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.853590 16269 raft_consensus.cc:348] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.853706 16269 raft_consensus.cc:374] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.853739 16269 raft_consensus.cc:729] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8020209fcc454974a566bd40e399e1d9, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.853858 16269 consensus_queue.cc:260] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.854097 16269 ts_tablet_manager.cc:1428] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:03:55.854699 15463 catalog_manager.cc:4841] TS b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259): tablet e9ac2b5906c84290a4def51ccb867c34 (table test-table [id=b54a4eee9c4b4f20a54c76828dcdc654]) successfully deleted
I20260806 16:03:55.855857 16266 tablet_bootstrap.cc:492] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: No bootstrap required, opened a new log
I20260806 16:03:55.855938 16266 ts_tablet_manager.cc:1397] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 16:03:55.856328 16306 ts_tablet_manager.cc:1905] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:55.856369 16266 raft_consensus.cc:348] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.856472 16266 raft_consensus.cc:374] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.856511 16266 raft_consensus.cc:729] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31ae33e599b4e438fb48e8843bb0aad, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.856623 16266 consensus_queue.cc:260] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.856869 16266 ts_tablet_manager.cc:1428] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:03:55.860774 16306 ts_tablet_manager.cc:1918] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:03:55.860862 16306 log.cc:1196] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/e9ac2b5906c84290a4def51ccb867c34
I20260806 16:03:55.861264 16306 ts_tablet_manager.cc:1939] T e9ac2b5906c84290a4def51ccb867c34 P 8020209fcc454974a566bd40e399e1d9: Deleting consensus metadata
I20260806 16:03:55.862529 16307 tablet_replica.cc:330] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: stopping tablet replica
I20260806 16:03:55.862672 15464 catalog_manager.cc:4841] TS 8020209fcc454974a566bd40e399e1d9 (127.13.148.3:45969): tablet e9ac2b5906c84290a4def51ccb867c34 (table test-table [id=b54a4eee9c4b4f20a54c76828dcdc654]) successfully deleted
I20260806 16:03:55.863397 16307 raft_consensus.cc:2229] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:03:55.863699 16265 tablet_bootstrap.cc:492] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: No bootstrap required, opened a new log
I20260806 16:03:55.863731 16307 raft_consensus.cc:2258] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:03:55.863821 16265 ts_tablet_manager.cc:1397] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260806 16:03:55.864279 16265 raft_consensus.cc:348] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.864405 16265 raft_consensus.cc:374] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.864445 16265 raft_consensus.cc:729] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fe39c336ce48c48672666a3970fcaa, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.864599 16265 consensus_queue.cc:260] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.864873 16265 ts_tablet_manager.cc:1428] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 16:03:55.865069 16307 ts_tablet_manager.cc:1905] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:55.868413 16307 ts_tablet_manager.cc:1918] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:03:55.868517 16307 log.cc:1196] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/e9ac2b5906c84290a4def51ccb867c34
I20260806 16:03:55.869081 16307 ts_tablet_manager.cc:1939] T e9ac2b5906c84290a4def51ccb867c34 P a7fe39c336ce48c48672666a3970fcaa: Deleting consensus metadata
I20260806 16:03:55.870333 15462 catalog_manager.cc:4841] TS a7fe39c336ce48c48672666a3970fcaa (127.13.148.1:46669): tablet e9ac2b5906c84290a4def51ccb867c34 (table test-table [id=b54a4eee9c4b4f20a54c76828dcdc654]) successfully deleted
I20260806 16:03:55.912746 16274 raft_consensus.cc:482] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:03:55.912907 16274 raft_consensus.cc:504] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.914251 16274 leader_election.cc:288] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259), 8020209fcc454974a566bd40e399e1d9 (127.13.148.3:45969)
I20260806 16:03:55.919787 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.914678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:52301 (local address 127.13.148.3:45969)
0806 16:03:55.914846 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.914852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.915166 (+ 314us) server_negotiation.cc:407] Connection header received
0806 16:03:55.915407 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.915410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.915472 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.915561 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:55.915960 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.916531 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.918667 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.919117 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.919266 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.919335 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.919560 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.919643 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:03:55.919924 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.914977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:50945 (local address 127.13.148.2:45259)
0806 16:03:55.915105 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.915110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.915126 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:03:55.915304 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.915307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.915359 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.915454 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:55.915791 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.916579 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.918640 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.919264 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.919384 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.919434 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.919694 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.919777 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:03:55.919838 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.914534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45969 (local address 127.13.148.1:52301)
0806 16:03:55.915083 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.915098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.915236 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.915677 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.915680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.915699 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:55.915840 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.915846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.916661 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.916668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.918453 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.918495 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.919262 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.919266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.919412 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.919449 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.919535 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.919652 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":121,"thread_start_us":88,"threads_started":1}
I20260806 16:03:55.920477 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.914473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45259 (local address 127.13.148.1:50945)
0806 16:03:55.915037 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.915054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.915172 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.915472 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.915476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.915495 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:55.915663 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.915671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.916713 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.916717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.918453 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.918495 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.919415 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.919419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.919539 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.919562 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.920228 (+ 666us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.920311 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":150,"thread_start_us":120,"threads_started":1}
I20260806 16:03:55.920595 16167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aeb9871aa447a2b82a44ddfcd45416" candidate_uuid: "a7fe39c336ce48c48672666a3970fcaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8020209fcc454974a566bd40e399e1d9" is_pre_election: true
I20260806 16:03:55.920897 16167 raft_consensus.cc:2454] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7fe39c336ce48c48672666a3970fcaa in term 0.
I20260806 16:03:55.920990 15950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aeb9871aa447a2b82a44ddfcd45416" candidate_uuid: "a7fe39c336ce48c48672666a3970fcaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ae33e599b4e438fb48e8843bb0aad" is_pre_election: true
I20260806 16:03:55.921157 15950 raft_consensus.cc:2454] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7fe39c336ce48c48672666a3970fcaa in term 0.
I20260806 16:03:55.921420 15630 leader_election.cc:302] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8020209fcc454974a566bd40e399e1d9, a7fe39c336ce48c48672666a3970fcaa; no voters:
I20260806 16:03:55.921797 16274 raft_consensus.cc:2781] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:03:55.921860 16274 raft_consensus.cc:482] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:03:55.921902 16274 raft_consensus.cc:3037] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:55.923309 16274 raft_consensus.cc:504] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.923786 16274 leader_election.cc:288] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [CANDIDATE]: Term 1 election: Requested vote from peers b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259), 8020209fcc454974a566bd40e399e1d9 (127.13.148.3:45969)
I20260806 16:03:55.924340 16167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aeb9871aa447a2b82a44ddfcd45416" candidate_uuid: "a7fe39c336ce48c48672666a3970fcaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8020209fcc454974a566bd40e399e1d9"
I20260806 16:03:55.924476 16167 raft_consensus.cc:3037] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:55.926004 16167 raft_consensus.cc:2454] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7fe39c336ce48c48672666a3970fcaa in term 1.
I20260806 16:03:55.926431 15630 leader_election.cc:302] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8020209fcc454974a566bd40e399e1d9, a7fe39c336ce48c48672666a3970fcaa; no voters:
I20260806 16:03:55.926738 16274 raft_consensus.cc:2781] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:55.926973 16274 raft_consensus.cc:686] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 1 LEADER]: Becoming Leader. State: Replica: a7fe39c336ce48c48672666a3970fcaa, State: Running, Role: LEADER
I20260806 16:03:55.927254 16274 consensus_queue.cc:237] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.930311 15490 catalog_manager.cc:5495] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa reported cstate change: term changed from 0 to 1, leader changed from <none> to a7fe39c336ce48c48672666a3970fcaa (127.13.148.1). New cstate: current_term: 1 leader_uuid: "a7fe39c336ce48c48672666a3970fcaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } health_report { overall_health: UNKNOWN } } }
I20260806 16:03:55.932761 15950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87aeb9871aa447a2b82a44ddfcd45416" candidate_uuid: "a7fe39c336ce48c48672666a3970fcaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ae33e599b4e438fb48e8843bb0aad"
I20260806 16:03:55.932910 15950 raft_consensus.cc:3037] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:55.934177 15950 raft_consensus.cc:2454] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7fe39c336ce48c48672666a3970fcaa in term 1.
Aug 06 16:03:55 dist-test-slave-7kzw krb5kdc[15421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032234, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:55.966554 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.956478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.13.148.62:35543)
0806 16:03:55.956642 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:55.956647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:55.956845 (+ 198us) server_negotiation.cc:407] Connection header received
0806 16:03:55.956987 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:55.956989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:55.957033 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:55.957124 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:55.958016 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.958633 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.959746 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.959980 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.960077 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.962951 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:55.962960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:55.962963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:55.962985 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:55.964581 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.965122 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.965126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.965128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.965213 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:55.965526 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:55.965532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:55.965534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:55.965733 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:55.965849 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:55.966189 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 16:03:55.966372 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":73}
I20260806 16:03:55.966555 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:55.956397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35543 (local address 127.0.0.1:52002)
0806 16:03:55.956737 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:55.956753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:55.956884 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:55.957180 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:55.957183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:55.957724 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:55.957852 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.957862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.958745 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.958749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.959581 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:55.959593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:55.960130 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:55.960134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:55.960241 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:55.961052 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:55.961066 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:55.962799 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:55.964720 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.964734 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.964741 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.965000 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.965320 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:55.965323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:55.965325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:55.965428 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:55.965861 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:55.965871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:55.965992 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 16:03:55.966119 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 16:03:55.966307 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":121,"threads_started":1}
I20260806 16:03:55.971333 15492 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52002:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 16:03:55.972172 15492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:03:55.977825 15702 tablet_service.cc:1467] Processing CreateTablet for tablet d380a552d15e41bb90c4f2eb380a57fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f81b92f586cf4d6d9fd0bcbf1a4d8429]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:03:55.977929 15917 tablet_service.cc:1467] Processing CreateTablet for tablet d380a552d15e41bb90c4f2eb380a57fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f81b92f586cf4d6d9fd0bcbf1a4d8429]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:03:55.978103 16134 tablet_service.cc:1467] Processing CreateTablet for tablet d380a552d15e41bb90c4f2eb380a57fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f81b92f586cf4d6d9fd0bcbf1a4d8429]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:03:55.978384 15702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d380a552d15e41bb90c4f2eb380a57fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.978381 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d380a552d15e41bb90c4f2eb380a57fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.978545 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d380a552d15e41bb90c4f2eb380a57fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:55.982002 16266 tablet_bootstrap.cc:492] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: Bootstrap starting.
I20260806 16:03:55.982661 16265 tablet_bootstrap.cc:492] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: Bootstrap starting.
I20260806 16:03:55.983059 16266 tablet_bootstrap.cc:654] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.983407 16269 tablet_bootstrap.cc:492] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: Bootstrap starting.
I20260806 16:03:55.983667 16265 tablet_bootstrap.cc:654] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.984335 16269 tablet_bootstrap.cc:654] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:55.985054 16265 tablet_bootstrap.cc:492] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: No bootstrap required, opened a new log
I20260806 16:03:55.985154 16265 ts_tablet_manager.cc:1397] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:03:55.985687 16265 raft_consensus.cc:348] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.985826 16265 raft_consensus.cc:374] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.985872 16265 raft_consensus.cc:729] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fe39c336ce48c48672666a3970fcaa, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.985985 16266 tablet_bootstrap.cc:492] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: No bootstrap required, opened a new log
I20260806 16:03:55.986059 16265 consensus_queue.cc:260] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.986145 16266 ts_tablet_manager.cc:1397] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 16:03:55.986435 16315 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:55.986632 16315 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:55.986418 16265 ts_tablet_manager.cc:1428] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:03:55.986650 16266 raft_consensus.cc:348] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.986763 16266 raft_consensus.cc:374] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.986795 16266 raft_consensus.cc:729] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31ae33e599b4e438fb48e8843bb0aad, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.986914 16266 consensus_queue.cc:260] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.987200 16266 ts_tablet_manager.cc:1428] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 16:03:55.987432 16275 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:55.987576 16275 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:55.987577 16269 tablet_bootstrap.cc:492] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: No bootstrap required, opened a new log
I20260806 16:03:55.987668 16269 ts_tablet_manager.cc:1397] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 16:03:55.988126 16269 raft_consensus.cc:348] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.988278 16269 raft_consensus.cc:374] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:55.988322 16269 raft_consensus.cc:729] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8020209fcc454974a566bd40e399e1d9, State: Initialized, Role: FOLLOWER
I20260806 16:03:55.988453 16269 consensus_queue.cc:260] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:55.988736 16269 ts_tablet_manager.cc:1428] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:03:55.988969 16300 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:55.989131 16300 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 16:03:56.077112 16235 tablet.cc:2369] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:03:56.080986 16019 tablet.cc:2369] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:03:56.087180 15796 tablet.cc:2369] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:03:56.228680 16300 raft_consensus.cc:482] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:03:56.228863 16300 raft_consensus.cc:504] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:56.229306 16300 leader_election.cc:288] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7fe39c336ce48c48672666a3970fcaa (127.13.148.1:46669), b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259)
I20260806 16:03:56.229911 15729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380a552d15e41bb90c4f2eb380a57fd" candidate_uuid: "8020209fcc454974a566bd40e399e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fe39c336ce48c48672666a3970fcaa" is_pre_election: true
I20260806 16:03:56.229911 15950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380a552d15e41bb90c4f2eb380a57fd" candidate_uuid: "8020209fcc454974a566bd40e399e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ae33e599b4e438fb48e8843bb0aad" is_pre_election: true
I20260806 16:03:56.230093 15950 raft_consensus.cc:2454] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8020209fcc454974a566bd40e399e1d9 in term 0.
I20260806 16:03:56.230093 15729 raft_consensus.cc:2454] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8020209fcc454974a566bd40e399e1d9 in term 0.
I20260806 16:03:56.230563 16060 leader_election.cc:302] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8020209fcc454974a566bd40e399e1d9, a7fe39c336ce48c48672666a3970fcaa, b31ae33e599b4e438fb48e8843bb0aad; no voters:
I20260806 16:03:56.230815 16300 raft_consensus.cc:2781] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:03:56.230881 16300 raft_consensus.cc:482] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:03:56.230909 16300 raft_consensus.cc:3037] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:56.232025 16300 raft_consensus.cc:504] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:56.232383 16300 leader_election.cc:288] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [CANDIDATE]: Term 1 election: Requested vote from peers a7fe39c336ce48c48672666a3970fcaa (127.13.148.1:46669), b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259)
I20260806 16:03:56.232823 15950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380a552d15e41bb90c4f2eb380a57fd" candidate_uuid: "8020209fcc454974a566bd40e399e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ae33e599b4e438fb48e8843bb0aad"
I20260806 16:03:56.232823 15729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d380a552d15e41bb90c4f2eb380a57fd" candidate_uuid: "8020209fcc454974a566bd40e399e1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7fe39c336ce48c48672666a3970fcaa"
I20260806 16:03:56.232967 15950 raft_consensus.cc:3037] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:56.232967 15729 raft_consensus.cc:3037] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:56.234164 15729 raft_consensus.cc:2454] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8020209fcc454974a566bd40e399e1d9 in term 1.
I20260806 16:03:56.234163 15950 raft_consensus.cc:2454] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8020209fcc454974a566bd40e399e1d9 in term 1.
I20260806 16:03:56.234570 16058 leader_election.cc:302] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8020209fcc454974a566bd40e399e1d9, a7fe39c336ce48c48672666a3970fcaa, b31ae33e599b4e438fb48e8843bb0aad; no voters:
I20260806 16:03:56.234778 16300 raft_consensus.cc:2781] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:56.234913 16300 raft_consensus.cc:686] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 8020209fcc454974a566bd40e399e1d9, State: Running, Role: LEADER
I20260806 16:03:56.235073 16300 consensus_queue.cc:237] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } }
I20260806 16:03:56.236248 16279 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: TxnStatusTablet state changed. Reason: New leader 8020209fcc454974a566bd40e399e1d9. Latest consensus state: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:56.236382 16279 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:03:56.236721 16333 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 16:03:56.236991 16333 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 16:03:56.237308 15492 catalog_manager.cc:5495] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8020209fcc454974a566bd40e399e1d9 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } health_report { overall_health: HEALTHY } } }
I20260806 16:03:56.289511 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.285850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.13.148.3:45969)
0806 16:03:56.285981 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:56.285986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:56.286003 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:03:56.286148 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:56.286151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:56.286205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:56.286289 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:03:56.286611 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.287199 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.288448 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.288655 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.288739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.289039 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:03:56.289107 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:03:56.289160 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:56.289351 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 16:03:56.289405 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:03:56.289510 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.285749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45969 (local address 127.0.0.1:35840)
0806 16:03:56.285907 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:56.285925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:56.286026 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:56.286314 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:56.286316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:56.286323 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:03:56.286417 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:56.286421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.287330 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.287333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:56.288321 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:56.288332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.288762 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.288765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:56.288855 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.288910 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:03:56.289190 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:03:56.289205 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 16:03:56.289274 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 16:03:56.289356 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 16:03:56.292419 15950 raft_consensus.cc:1261] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Refusing update from remote peer 8020209fcc454974a566bd40e399e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:56.292459 15729 raft_consensus.cc:1261] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Refusing update from remote peer 8020209fcc454974a566bd40e399e1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:56.292946 16300 consensus_queue.cc:1035] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:56.293145 16279 consensus_queue.cc:1035] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:56.295414 16315 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: TxnStatusTablet state changed. Reason: New leader 8020209fcc454974a566bd40e399e1d9. Latest consensus state: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:56.295531 16315 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:56.295637 16275 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: TxnStatusTablet state changed. Reason: New leader 8020209fcc454974a566bd40e399e1d9. Latest consensus state: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:56.295754 16275 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:56.296624 16279 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:56.296748 16279 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:03:56.297706 16315 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:56.297803 16315 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P a7fe39c336ce48c48672666a3970fcaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:56.301319 16277 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:56.301450 16277 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P 8020209fcc454974a566bd40e399e1d9: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:03:56.303478 16290 mvcc.cc:205] Tried to move back new op lower bound from 7315588039849361408 to 7315588039620812800. Current Snapshot: MvccSnapshot[applied={T|T < 7315588039849361408 or (T in {7315588039849361408})}]
I20260806 16:03:56.303727 16275 tablet_replica.cc:439] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8020209fcc454974a566bd40e399e1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fe39c336ce48c48672666a3970fcaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46669 } } peers { permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 } } peers { permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 } } }
I20260806 16:03:56.303840 16275 tablet_replica.cc:442] T d380a552d15e41bb90c4f2eb380a57fd P b31ae33e599b4e438fb48e8843bb0aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:56.310658 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.305862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.13.148.1:46669)
0806 16:03:56.306397 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:56.306403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:56.306414 (+ 11us) server_negotiation.cc:407] Connection header received
0806 16:03:56.306537 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:56.306540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:56.306613 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:56.306714 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:03:56.307044 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.307690 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.308939 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.309196 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.309282 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.309645 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:03:56.310115 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:03:56.310276 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:56.310366 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 16:03:56.310422 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":95,"threads_started":1}
I20260806 16:03:56.322328 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.318334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45969 (local address 127.13.148.1:46397)
0806 16:03:56.318723 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:56.318740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:56.318890 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:56.319211 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:56.319215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:56.319223 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:03:56.319377 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:56.319383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.320202 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.320208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:56.320936 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:56.320945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.321458 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.321462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:56.321588 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.321634 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:03:56.321975 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:03:56.321985 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 16:03:56.322079 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 16:03:56.322164 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":134,"threads_started":1}
I20260806 16:03:56.322315 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.318525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:46397 (local address 127.13.148.3:45969)
0806 16:03:56.318682 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:56.318687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:56.318839 (+ 152us) server_negotiation.cc:407] Connection header received
0806 16:03:56.319022 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:56.319025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:56.319085 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:56.319201 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:03:56.319502 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.320052 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.321077 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.321329 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.321455 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.321801 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:03:56.321881 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:03:56.321971 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:56.322094 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 16:03:56.322159 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:03:56.330786 16167 raft_consensus.cc:1261] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Refusing update from remote peer a7fe39c336ce48c48672666a3970fcaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:56.330914 15950 raft_consensus.cc:1261] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Refusing update from remote peer a7fe39c336ce48c48672666a3970fcaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:56.331321 16315 consensus_queue.cc:1035] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8020209fcc454974a566bd40e399e1d9" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:56.331785 16315 consensus_queue.cc:1035] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31ae33e599b4e438fb48e8843bb0aad" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:56.360369 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.355739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46669 (local address 127.13.148.3:50111)
0806 16:03:56.356048 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:56.356066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:56.356172 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:56.356845 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:56.356848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:56.356857 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:03:56.357246 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:56.357254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.358107 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.358112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:56.358855 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:56.358895 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.359536 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.359539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:56.359608 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.359650 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:03:56.360052 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:03:56.360066 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 16:03:56.360150 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 16:03:56.360234 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260806 16:03:56.360369 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.356021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50111 (local address 127.13.148.1:46669)
0806 16:03:56.356608 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:56.356612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:56.356628 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:03:56.356677 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:56.356680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:56.356735 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:56.356816 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:03:56.357382 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.357990 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.359033 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.359335 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.359421 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.359800 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:03:56.359943 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:03:56.360015 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:56.360164 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 16:03:56.360220 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503}
I20260806 16:03:56.388553 15492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:03:56.388772 15492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:03:56.391392 15492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24321c420d264e318431340b2cbcf697: Sending DeleteTablet for 3 replicas of tablet 87aeb9871aa447a2b82a44ddfcd45416
I20260806 16:03:56.392210 15702 tablet_service.cc:1514] Processing DeleteTablet for tablet 87aeb9871aa447a2b82a44ddfcd45416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:56 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51344
I20260806 16:03:56.392508 15917 tablet_service.cc:1514] Processing DeleteTablet for tablet 87aeb9871aa447a2b82a44ddfcd45416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:56 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:40624
I20260806 16:03:56.392966 16357 tablet_replica.cc:330] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: stopping tablet replica
I20260806 16:03:56.393132 16357 raft_consensus.cc:2229] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:03:56.393404 16357 raft_consensus.cc:2258] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:03:56.393687 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 87aeb9871aa447a2b82a44ddfcd45416 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:56 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:35822
I20260806 16:03:56.394115 16357 ts_tablet_manager.cc:1905] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:56.394625 16358 tablet_replica.cc:330] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: stopping tablet replica
I20260806 16:03:56.394749 16358 raft_consensus.cc:2229] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:03:56.394857 16358 raft_consensus.cc:2258] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:03:56.395324 16364 tablet_replica.cc:330] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: stopping tablet replica
I20260806 16:03:56.395465 16358 ts_tablet_manager.cc:1905] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:56.395519 16364 raft_consensus.cc:2229] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:03:56.395766 16364 raft_consensus.cc:2258] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:03:56.396474 16364 ts_tablet_manager.cc:1905] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:56.398526 16358 ts_tablet_manager.cc:1918] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:03:56.398613 16358 log.cc:1196] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/87aeb9871aa447a2b82a44ddfcd45416
I20260806 16:03:56.398862 16358 ts_tablet_manager.cc:1939] T 87aeb9871aa447a2b82a44ddfcd45416 P b31ae33e599b4e438fb48e8843bb0aad: Deleting consensus metadata
I20260806 16:03:56.399633 15463 catalog_manager.cc:4841] TS b31ae33e599b4e438fb48e8843bb0aad (127.13.148.2:45259): tablet 87aeb9871aa447a2b82a44ddfcd45416 (table test-table [id=b0ff086fa5e54173975bd900b3a74913]) successfully deleted
I20260806 16:03:56.400038 16364 ts_tablet_manager.cc:1918] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:03:56.400194 16364 log.cc:1196] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/87aeb9871aa447a2b82a44ddfcd45416
I20260806 16:03:56.400563 16364 ts_tablet_manager.cc:1939] T 87aeb9871aa447a2b82a44ddfcd45416 P 8020209fcc454974a566bd40e399e1d9: Deleting consensus metadata
I20260806 16:03:56.401780 15464 catalog_manager.cc:4841] TS 8020209fcc454974a566bd40e399e1d9 (127.13.148.3:45969): tablet 87aeb9871aa447a2b82a44ddfcd45416 (table test-table [id=b0ff086fa5e54173975bd900b3a74913]) successfully deleted
I20260806 16:03:56.402149 16357 ts_tablet_manager.cc:1918] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:03:56.402266 16357 log.cc:1196] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/87aeb9871aa447a2b82a44ddfcd45416
I20260806 16:03:56.402607 16357 ts_tablet_manager.cc:1939] T 87aeb9871aa447a2b82a44ddfcd45416 P a7fe39c336ce48c48672666a3970fcaa: Deleting consensus metadata
I20260806 16:03:56.403302 15462 catalog_manager.cc:4841] TS a7fe39c336ce48c48672666a3970fcaa (127.13.148.1:46669): tablet 87aeb9871aa447a2b82a44ddfcd45416 (table test-table [id=b0ff086fa5e54173975bd900b3a74913]) successfully deleted
Aug 06 16:03:56 dist-test-slave-7kzw krb5kdc[15421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:03:56.416018 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.394499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.13.148.1:46669)
0806 16:03:56.394627 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:56.394632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:56.400966 (+ 6334us) server_negotiation.cc:407] Connection header received
0806 16:03:56.402244 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:56.402248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:56.402307 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:56.402365 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:56.403490 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.404387 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.405804 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.406071 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.406183 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.411112 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:56.411126 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:56.411149 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:56.411187 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:56.413580 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:56.414124 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:56.414131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:56.414134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:56.414246 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:56.414670 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:56.414674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:56.414676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:56.415058 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:56.415321 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:56.415548 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 16:03:56.415702 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 16:03:56.416829 15788 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51356
I20260806 16:03:56.427784 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:56.418800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.13.148.62:35543)
0806 16:03:56.418951 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:56.418955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:56.419016 (+ 61us) server_negotiation.cc:407] Connection header received
0806 16:03:56.419160 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:56.419162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:56.419205 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:56.419288 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:56.420149 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.420993 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.422342 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:56.422714 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:56.422801 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:56.424540 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:56.424543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:56.424545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:56.424571 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:56.425968 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:56.426445 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:56.426448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:56.426449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:56.426597 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:56.426892 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:56.426894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:56.426895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:56.427072 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:56.427172 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:56.427441 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 16:03:56.427540 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 16:03:56.428287 15492 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52012
I20260806 16:03:56.429976 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 15602
I20260806 16:03:56.442555 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 15809
I20260806 16:03:56.453645 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 16031
I20260806 16:03:56.464828 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 15438
2026-08-06T16:03:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3277 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:03:56 dist-test-slave-7kzw krb5kdc[16381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:03:56 dist-test-slave-7kzw krb5kdc[16381](info): set up 2 sockets
Aug 06 16:03:56 dist-test-slave-7kzw krb5kdc[16381](info): commencing operation
krb5kdc: starting...
W20260806 16:03:57.731135 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:03:57 dist-test-slave-7kzw krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:03:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:03:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:03:57.866828 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39165
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:57.988278 16397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:57.988555 16397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:57.988597 16397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:57.992568 16397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:03:57.992628 16397 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 16:03:57.992646 16397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:57.992663 16397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:57.992681 16397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:03:57.992695 16397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:03:57.998250 16397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39165
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:57.999405 16397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:58.000550 16397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:58.009603 16407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:58.009564 16403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:58.009595 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:58.009894 16397 server_base.cc:1034] running on GCE node
I20260806 16:03:58.010686 16397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:58.011785 16397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:58.012956 16397 hybrid_clock.cc:648] HybridClock initialized: now 1786032238012922 us; error 47 us; skew 500 ppm
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:58.021240 16397 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:03:58.022373 16397 webserver.cc:460] Webserver started at http://127.13.148.62:33805/ using document root <none> and password file <none>
I20260806 16:03:58.023046 16397 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:58.023154 16397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:58.023422 16397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:58.025738 16397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "a2aaaf5400dd4993bef0eaaacac48c99"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "57734ef2619bea1afa0066846ca8f4fc"
server_key_iv: "f9566f1a669b2582ce09a81088753ba3"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.026376 16397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "a2aaaf5400dd4993bef0eaaacac48c99"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "57734ef2619bea1afa0066846ca8f4fc"
server_key_iv: "f9566f1a669b2582ce09a81088753ba3"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.030587 16397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:03:58.033289 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:58.034675 16397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:58.034822 16397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "a2aaaf5400dd4993bef0eaaacac48c99"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "57734ef2619bea1afa0066846ca8f4fc"
server_key_iv: "f9566f1a669b2582ce09a81088753ba3"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.034942 16397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:58.048488 16397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:58.051939 16397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:58.052178 16397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:58.060961 16397 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39165
I20260806 16:03:58.060988 16501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39165 every 8 connection(s)
I20260806 16:03:58.061976 16397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:03:58.064996 16502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:58.065429 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 16397
I20260806 16:03:58.065562 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:03:58.065842 13904 external_mini_cluster.cc:1428] Setting key 7d5964d84bb1c030d02a4cae4682ded6
I20260806 16:03:58.073014 16502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: Bootstrap starting.
I20260806 16:03:58.075963 16502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: Neither blocks nor log segments found. Creating new log.
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:58.076907 16502 log.cc:824] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:58.079871 16502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: No bootstrap required, opened a new log
I20260806 16:03:58.083460 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.067417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.13.148.62:39165)
0806 16:03:58.067878 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.067899 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.067947 (+ 48us) server_negotiation.cc:407] Connection header received
0806 16:03:58.068812 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.068839 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.069314 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.069714 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:58.070407 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.072137 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.073017 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.073761 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.073941 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.076818 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:58.076824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:58.076843 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:58.076877 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:58.080049 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.080498 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.080505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.080511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.080668 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.080958 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.080961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.080963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.081507 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:58.081717 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.082019 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.082362 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":126,"threads_started":1}
I20260806 16:03:58.084215 16502 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2aaaf5400dd4993bef0eaaacac48c99" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39165 } }
I20260806 16:03:58.084455 16502 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:58.084522 16502 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2aaaf5400dd4993bef0eaaacac48c99, State: Initialized, Role: FOLLOWER
I20260806 16:03:58.085065 16502 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2aaaf5400dd4993bef0eaaacac48c99" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39165 } }
I20260806 16:03:58.085240 16502 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:03:58.085314 16502 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:03:58.085421 16502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:58.086457 16502 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2aaaf5400dd4993bef0eaaacac48c99" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39165 } }
I20260806 16:03:58.086900 16502 leader_election.cc:302] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2aaaf5400dd4993bef0eaaacac48c99; no voters:
I20260806 16:03:58.087248 16502 leader_election.cc:288] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:03:58.087435 16508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:58.087751 16508 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [term 1 LEADER]: Becoming Leader. State: Replica: a2aaaf5400dd4993bef0eaaacac48c99, State: Running, Role: LEADER
I20260806 16:03:58.088127 16508 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2aaaf5400dd4993bef0eaaacac48c99" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39165 } }
I20260806 16:03:58.088788 16502 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:03:58.090348 16509 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2aaaf5400dd4993bef0eaaacac48c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2aaaf5400dd4993bef0eaaacac48c99" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39165 } } }
I20260806 16:03:58.090507 16509 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [sys.catalog]: This master's current role is: LEADER
I20260806 16:03:58.090306 16510 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2aaaf5400dd4993bef0eaaacac48c99. Latest consensus state: current_term: 1 leader_uuid: "a2aaaf5400dd4993bef0eaaacac48c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2aaaf5400dd4993bef0eaaacac48c99" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39165 } } }
I20260806 16:03:58.090828 16510 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99 [sys.catalog]: This master's current role is: LEADER
W20260806 16:03:58.094010 16528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:03:58.094089 16528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:03:58.094151 16523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:03:58.095094 16523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:03:58.101624 16523 catalog_manager.cc:1347] Generated new cluster ID: f2f735d73c8847c2a33163cc7c853aa2
I20260806 16:03:58.101790 16523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:03:58.142405 16523 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:03:58.143456 16523 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:03:58.155103 16523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: Generated new TSK 0
I20260806 16:03:58.155891 16523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:03:58.197587 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39165
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:58.314234 16538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:58.314596 16538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:58.314710 16538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:58.318764 16538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:58.318830 16538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:58.318869 16538 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 16:03:58.318883 16538 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:03:58.318958 16538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:03:58.324316 16538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:39165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:58.325397 16538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:58.326493 16538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:58.336647 16544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:58.336665 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:58.336841 16538 server_base.cc:1034] running on GCE node
W20260806 16:03:58.336752 16545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:58.337365 16538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:58.338254 16538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:58.339465 16538 hybrid_clock.cc:648] HybridClock initialized: now 1786032238339445 us; error 65 us; skew 500 ppm
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:58.342648 16538 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:03:58.343530 16538 webserver.cc:460] Webserver started at http://127.13.148.1:35529/ using document root <none> and password file <none>
I20260806 16:03:58.344075 16538 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:58.344138 16538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:58.344322 16538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:58.346450 16538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "b5456ab5278b40488b27912b3556de04"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "a11679a452070df1b92d5729af641007"
server_key_iv: "bb80be583df5bf491c02c6eb67fbeb98"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.347074 16538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "b5456ab5278b40488b27912b3556de04"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "a11679a452070df1b92d5729af641007"
server_key_iv: "bb80be583df5bf491c02c6eb67fbeb98"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.351222 16538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 16:03:58.354038 16557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:58.355924 16538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:03:58.356046 16538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "b5456ab5278b40488b27912b3556de04"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "a11679a452070df1b92d5729af641007"
server_key_iv: "bb80be583df5bf491c02c6eb67fbeb98"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.356164 16538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:58.370162 16538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:58.373513 16538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:58.373688 16538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:58.375288 16538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:58.375368 16538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:58.375442 16538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:58.375486 16538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:58.386605 16538 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:46595
I20260806 16:03:58.386746 16712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:46595 every 8 connection(s)
I20260806 16:03:58.387682 16538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:03:58.391185 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.376181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:34743 (local address 127.13.148.62:39165)
0806 16:03:58.376329 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.376333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.377490 (+ 1157us) server_negotiation.cc:407] Connection header received
0806 16:03:58.378678 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.378683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.378753 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.378819 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:58.380772 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.381705 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.383526 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.383790 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.383918 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.386978 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:58.386981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:58.386983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:58.387010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:58.388874 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.389497 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.389520 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.389524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.389640 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.390079 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.390082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.390083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.390281 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:58.390359 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.390925 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.391042 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:03:58.392005 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.376710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39165 (local address 127.13.148.1:34743)
0806 16:03:58.377309 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.377355 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.378354 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.379302 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.379311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.379814 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:58.380538 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.380619 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.381859 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.381863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.383384 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.383396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.383959 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.383963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.384157 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.384890 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:58.384906 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:58.386763 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:58.389003 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.389008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.389019 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.389368 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.389758 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.389761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.389762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.389959 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.390402 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:58.390411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:58.390690 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.390845 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.391105 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260806 16:03:58.393437 16714 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39165
I20260806 16:03:58.393587 16714 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:58.394224 16714 heartbeater.cc:511] Master 127.13.148.62:39165 requested a full tablet report, sending...
I20260806 16:03:58.396044 16439 ts_manager.cc:194] Registered new tserver with Master: b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595)
I20260806 16:03:58.396130 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 16538
I20260806 16:03:58.396238 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:03:58.396548 13904 external_mini_cluster.cc:1428] Setting key 8b3c538e782d27db93077d03854e3a2d
I20260806 16:03:58.397636 16439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:34743
I20260806 16:03:58.404896 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.395963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:50895 (local address 127.13.148.62:39165)
0806 16:03:58.396082 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.396087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.397155 (+ 1068us) server_negotiation.cc:407] Connection header received
0806 16:03:58.397258 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.397260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.397299 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.397365 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:58.397891 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.398565 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.399641 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.399944 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.400055 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.401443 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:58.401446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:58.401448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:58.401471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:58.403239 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.403676 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.403681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.403683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.403769 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.404051 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.404055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.404057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.404244 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:58.404355 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.404617 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.404731 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:03:58.404917 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.396051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39165 (local address 127.13.148.1:50895)
0806 16:03:58.397062 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.397077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.397177 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.397405 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.397408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.397622 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:58.397767 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.397776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.398703 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.398707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.399517 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.399528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.400079 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.400086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.400204 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.400772 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:58.400792 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:58.401293 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:58.403364 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.403367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.403368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.403565 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.403864 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.403868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.403870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.403958 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.404345 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:58.404348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:58.404446 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.404551 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.404731 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":155,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:03:58.428226 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39165
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:58.548202 16728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:58.548488 16728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:58.548538 16728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:58.552695 16728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:58.552764 16728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:58.552806 16728 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 16:03:58.552850 16728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:03:58.552914 16728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:03:58.558607 16728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:39165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:58.559768 16728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:58.560937 16728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:58.571473 16736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:58.571593 16738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:58.571471 16734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:58.571715 16728 server_base.cc:1034] running on GCE node
I20260806 16:03:58.572274 16728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:58.572971 16728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:58.574170 16728 hybrid_clock.cc:648] HybridClock initialized: now 1786032238574158 us; error 30 us; skew 500 ppm
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:58.577170 16728 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:03:58.578150 16728 webserver.cc:460] Webserver started at http://127.13.148.2:44877/ using document root <none> and password file <none>
I20260806 16:03:58.578794 16728 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:58.578862 16728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:58.579048 16728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:58.581076 16728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "48f337453cdf494995511f074951c8fa"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "678ea59374358ef7c4fd0c564b4a6f09"
server_key_iv: "503ff0c78c9e375e9f564989ab2cf0be"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.581615 16728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "48f337453cdf494995511f074951c8fa"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "678ea59374358ef7c4fd0c564b4a6f09"
server_key_iv: "503ff0c78c9e375e9f564989ab2cf0be"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.585439 16728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 16:03:58.588104 16749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:58.589486 16728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:03:58.589597 16728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "48f337453cdf494995511f074951c8fa"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "678ea59374358ef7c4fd0c564b4a6f09"
server_key_iv: "503ff0c78c9e375e9f564989ab2cf0be"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.589756 16728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:58.602432 16728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:58.605933 16728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:58.606173 16728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:58.607892 16728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:58.607949 16728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:58.607995 16728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:58.608031 16728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:58.619261 16728 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:35117
I20260806 16:03:58.619335 16911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:35117 every 8 connection(s)
I20260806 16:03:58.620306 16728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:03:58.622328 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.608624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:59565 (local address 127.13.148.62:39165)
0806 16:03:58.608789 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.608794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.609720 (+ 926us) server_negotiation.cc:407] Connection header received
0806 16:03:58.610830 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.610835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.610903 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.610991 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:58.612456 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.613020 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.614174 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.614564 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.614683 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.618172 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:58.618175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:58.618178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:58.618215 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:58.619916 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.620576 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.620580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.620581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.620709 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.621168 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.621175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.621178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.621430 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:58.621515 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.622067 (+ 552us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.622204 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:03:58.623255 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.609064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39165 (local address 127.13.148.2:59565)
0806 16:03:58.609540 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.609592 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.610553 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.611361 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.611368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.611712 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:58.612289 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.612312 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.613155 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.613159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.614042 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.614051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.614743 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.614747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.614977 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.615757 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:58.615773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:58.617991 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:58.620036 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.620042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.620056 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.620434 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.620822 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.620826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.620828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.621012 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.621563 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:58.621569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:58.621820 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.621991 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.622269 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
I20260806 16:03:58.624632 16912 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39165
I20260806 16:03:58.624809 16912 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:58.625335 16912 heartbeater.cc:511] Master 127.13.148.62:39165 requested a full tablet report, sending...
I20260806 16:03:58.626492 16439 ts_manager.cc:194] Registered new tserver with Master: 48f337453cdf494995511f074951c8fa (127.13.148.2:35117)
I20260806 16:03:58.627127 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 16728
I20260806 16:03:58.627303 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:03:58.627327 16439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:59565
I20260806 16:03:58.627638 13904 external_mini_cluster.cc:1428] Setting key 4da48fb95e1fa4ddeed7267c61604523
I20260806 16:03:58.636006 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.626917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39165 (local address 127.13.148.2:39151)
0806 16:03:58.627212 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.627227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.627334 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.627864 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.627867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.628097 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:58.628247 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.628256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.629269 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.629276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.630222 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.630237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.630884 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.630891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.631022 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.631779 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:58.631791 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:58.632112 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:58.634230 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.634232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.634234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.634422 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.634834 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.634837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.634839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.634943 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.635519 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:58.635522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:58.635637 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.635734 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.635851 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
I20260806 16:03:58.636238 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.626980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39151 (local address 127.13.148.62:39165)
0806 16:03:58.627634 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.627637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.627674 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:03:58.627710 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.627712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.627752 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.627834 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:58.628380 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.629117 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.630373 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.630747 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.630859 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.632265 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:58.632270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:58.632273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:58.632304 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:58.634102 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.634594 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.634600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.634605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.634708 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.635043 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.635049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.635054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.635394 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:58.635508 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.635840 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.635991 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598}
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:03:58.661491 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39165
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:03:58.779470 16928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:03:58.779806 16928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:03:58.779897 16928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:03:58.783965 16928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:03:58.784076 16928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:03:58.784155 16928 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 16:03:58.784201 16928 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:03:58.784276 16928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:03:58.789988 16928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:39165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:03:58.791169 16928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:03:58.792255 16928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:03:58.802953 16938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:03:58.803093 16936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:58.803035 16928 server_base.cc:1034] running on GCE node
W20260806 16:03:58.802956 16940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:03:58.803799 16928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:03:58.804482 16928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:03:58.805691 16928 hybrid_clock.cc:648] HybridClock initialized: now 1786032238805667 us; error 34 us; skew 500 ppm
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:03:58.808761 16928 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:03:58.809620 16928 webserver.cc:460] Webserver started at http://127.13.148.3:35545/ using document root <none> and password file <none>
I20260806 16:03:58.810175 16928 fs_manager.cc:359] Metadata directory not provided
I20260806 16:03:58.810238 16928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:03:58.810411 16928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:03:58.812495 16928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "61edb70fbb7a4716bb47d6797b29306e"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "242a566476c92c80b3ba36a026bb0541"
server_key_iv: "a59597d021f3c26000594e2cd2cce85a"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.813014 16928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "61edb70fbb7a4716bb47d6797b29306e"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "242a566476c92c80b3ba36a026bb0541"
server_key_iv: "a59597d021f3c26000594e2cd2cce85a"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.816898 16928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:03:58.819522 16949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:58.821085 16928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:03:58.821221 16928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "61edb70fbb7a4716bb47d6797b29306e"
format_stamp: "Formatted at 2026-08-06 16:03:58 on dist-test-slave-7kzw"
server_key: "242a566476c92c80b3ba36a026bb0541"
server_key_iv: "a59597d021f3c26000594e2cd2cce85a"
server_key_version: "encryptionkey@0"
I20260806 16:03:58.821321 16928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:03:58.850040 16928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:03:58.853773 16928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:03:58.853950 16928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:03:58.855659 16928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:03:58.855748 16928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:03:58.855831 16928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:03:58.855876 16928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:58.868722 16928 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:43467
I20260806 16:03:58.868753 17121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:43467 every 8 connection(s)
I20260806 16:03:58.869900 16928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:03:58.871685 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.856561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:34621 (local address 127.13.148.62:39165)
0806 16:03:58.856704 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.856708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.857889 (+ 1181us) server_negotiation.cc:407] Connection header received
0806 16:03:58.859143 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.859174 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.859290 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.859380 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:58.860930 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.861718 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.863167 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.863508 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.863630 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.867217 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:58.867221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:58.867224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:58.867255 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:58.869284 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.869904 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.869908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.869913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.870019 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.870382 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.870386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.870388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.870682 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:58.870787 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.871373 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.871514 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:03:58.872511 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.857110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39165 (local address 127.13.148.3:34621)
0806 16:03:58.857716 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.857763 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.858904 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.859603 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.859610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.860104 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:58.860755 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.860773 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.861851 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.861855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.863025 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.863035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.863693 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.863699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.863888 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.864728 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:58.864745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:58.866932 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:58.869431 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.869439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.869457 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.869764 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.870107 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.870110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.870111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.870260 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.870798 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:58.870807 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:58.871110 (+ 303us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.871310 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.871684 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":145,"threads_started":1}
I20260806 16:03:58.873832 17123 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39165
I20260806 16:03:58.874054 17123 heartbeater.cc:464] Registering TS with master...
I20260806 16:03:58.874706 17123 heartbeater.cc:511] Master 127.13.148.62:39165 requested a full tablet report, sending...
I20260806 16:03:58.875983 16440 ts_manager.cc:194] Registered new tserver with Master: 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:03:58.876710 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:34621
I20260806 16:03:58.879914 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 16928
I20260806 16:03:58.880059 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:03:58.880410 13904 external_mini_cluster.cc:1428] Setting key 0e007c4e5ce306aa99901c8a0c912f6b
I20260806 16:03:58.883069 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:03:58.884955 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.876198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39165 (local address 127.13.148.3:38259)
0806 16:03:58.876617 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.876631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.876767 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.877116 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.877120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.877344 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:58.877504 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.877512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.878576 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.878579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.879254 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.879264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.879790 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.879795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.879907 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.880493 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:58.880511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:58.880941 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:58.883281 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.883284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.883286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.883517 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.883844 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:58.883846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:58.883848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:58.883938 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:58.884477 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:58.884480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:58.884574 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.884670 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.884797 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":122,"threads_started":1}
I20260806 16:03:58.885286 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.876316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:38259 (local address 127.13.148.62:39165)
0806 16:03:58.876457 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.876461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.876758 (+ 297us) server_negotiation.cc:407] Connection header received
0806 16:03:58.876894 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.876897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.876940 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.877035 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:58.877657 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.878383 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.879380 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.879642 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.879800 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.881105 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:58.881109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:58.881111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:58.881135 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:58.883116 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.883627 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.883630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.883634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.883739 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.884041 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.884058 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.884062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.884363 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:58.884541 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.884794 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.884997 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:03:58 dist-test-slave-7kzw krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:58.915838 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.906854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.13.148.62:39165)
0806 16:03:58.907006 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.907010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.907025 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:03:58.907071 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.907074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.907114 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.907210 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:58.908282 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.908902 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.909819 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.910064 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.910213 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.912887 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:58.912891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:58.912893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:58.912930 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:58.914238 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.914709 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.914712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.914713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.914832 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:58.915060 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:58.915062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:58.915063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:58.915240 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:58.915342 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.915523 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.915623 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:03:58.919355 16440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:03:58.922894 16440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:03:58.938817 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.933049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.13.148.2:35117)
0806 16:03:58.933391 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.933406 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.934258 (+ 852us) server_negotiation.cc:407] Connection header received
0806 16:03:58.934347 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.934353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.934610 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.934791 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:58.935105 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.935926 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.937407 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.938054 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.938189 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.938372 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.938543 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.938652 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":131,"threads_started":1}
I20260806 16:03:58.938832 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.932860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35117 (local address 127.0.0.1:54516)
0806 16:03:58.933908 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.933968 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.934107 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.934778 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.934785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.934808 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:58.934968 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.934976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.936051 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.936056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.937284 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.937294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.938179 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.938183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.938299 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.938339 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.938543 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.938696 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":3,"spinlock_wait_cycles":7168,"thread_start_us":73,"threads_started":1}
I20260806 16:03:58.939064 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.932870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46595 (local address 127.0.0.1:34186)
0806 16:03:58.933908 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.933968 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.934107 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.934957 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.934960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.934986 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:58.935111 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.935119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.936618 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.936621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.937569 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.937580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.938654 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.938658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.938772 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.938796 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.938844 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.938917 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":22,"spinlock_wait_cycles":5120,"thread_start_us":114,"threads_started":1}
I20260806 16:03:58.940227 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.933591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:43467 (local address 127.0.0.1:45556)
0806 16:03:58.934347 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:58.934359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:58.934432 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:58.934959 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:58.934962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:58.934993 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:58.935114 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.935120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.936563 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.936567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.938769 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:58.938778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.939803 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.939808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:58.939911 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.939937 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 16:03:58.940020 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 16:03:58.940098 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":3,"spinlock_wait_cycles":9728,"thread_start_us":92,"threads_started":1}
I20260806 16:03:58.940515 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.934000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.13.148.3:43467)
0806 16:03:58.934446 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.934457 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.934533 (+ 76us) server_negotiation.cc:407] Connection header received
0806 16:03:58.934613 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.934618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.934798 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.934944 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:58.935238 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.936426 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.938904 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.939675 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.939786 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.939942 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.940279 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.940377 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":127,"threads_started":1}
I20260806 16:03:58.941943 16828 tablet_service.cc:1467] Processing CreateTablet for tablet d56fb8704ff644518fc8b87de80cf1b6 (DEFAULT_TABLE table=test-table [id=88b67103b54f48c09e511293a5c633ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:58.943004 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56fb8704ff644518fc8b87de80cf1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:58.942739 17024 tablet_service.cc:1467] Processing CreateTablet for tablet d56fb8704ff644518fc8b87de80cf1b6 (DEFAULT_TABLE table=test-table [id=88b67103b54f48c09e511293a5c633ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:58.943252 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:58.933541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.13.148.1:46595)
0806 16:03:58.934522 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:58.934528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:58.934547 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:03:58.934613 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:58.934618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:58.934798 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:58.935299 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:58.935439 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.936506 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.937701 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:58.938524 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:58.942815 (+ 4291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:58.942965 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:58.943018 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 16:03:58.943113 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":102,"threads_started":1}
I20260806 16:03:58.943747 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56fb8704ff644518fc8b87de80cf1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:58.946125 16628 tablet_service.cc:1467] Processing CreateTablet for tablet d56fb8704ff644518fc8b87de80cf1b6 (DEFAULT_TABLE table=test-table [id=88b67103b54f48c09e511293a5c633ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:58.947357 16628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d56fb8704ff644518fc8b87de80cf1b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:58.950254 17151 tablet_bootstrap.cc:492] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Bootstrap starting.
I20260806 16:03:58.952992 17151 tablet_bootstrap.cc:654] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:58.953754 17151 log.cc:824] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:58.954507 17154 tablet_bootstrap.cc:492] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Bootstrap starting.
I20260806 16:03:58.955787 17151 tablet_bootstrap.cc:492] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: No bootstrap required, opened a new log
I20260806 16:03:58.955998 17151 ts_tablet_manager.cc:1397] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:03:58.956426 17154 tablet_bootstrap.cc:654] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:58.957118 17154 log.cc:824] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:58.954469 17152 tablet_bootstrap.cc:492] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Bootstrap starting.
I20260806 16:03:58.959136 17154 tablet_bootstrap.cc:492] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: No bootstrap required, opened a new log
I20260806 16:03:58.959348 17154 ts_tablet_manager.cc:1397] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:03:58.959560 17151 raft_consensus.cc:348] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:58.959889 17151 raft_consensus.cc:374] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:58.959965 17151 raft_consensus.cc:729] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f337453cdf494995511f074951c8fa, State: Initialized, Role: FOLLOWER
I20260806 16:03:58.960414 17151 consensus_queue.cc:260] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:58.960964 17152 tablet_bootstrap.cc:654] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:58.961141 16912 heartbeater.cc:503] Master 127.13.148.62:39165 was elected leader, sending a full tablet report...
I20260806 16:03:58.961489 17151 ts_tablet_manager.cc:1428] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260806 16:03:58.961989 17152 log.cc:824] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:03:58.964169 17152 tablet_bootstrap.cc:492] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: No bootstrap required, opened a new log
I20260806 16:03:58.963991 17154 raft_consensus.cc:348] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:58.964273 17154 raft_consensus.cc:374] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:58.964339 17154 raft_consensus.cc:729] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5456ab5278b40488b27912b3556de04, State: Initialized, Role: FOLLOWER
I20260806 16:03:58.964438 17152 ts_tablet_manager.cc:1397] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260806 16:03:58.964829 17154 consensus_queue.cc:260] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:58.965791 17154 ts_tablet_manager.cc:1428] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:03:58.966214 16714 heartbeater.cc:503] Master 127.13.148.62:39165 was elected leader, sending a full tablet report...
I20260806 16:03:58.967900 17152 raft_consensus.cc:348] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:58.968137 17152 raft_consensus.cc:374] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:58.968214 17152 raft_consensus.cc:729] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61edb70fbb7a4716bb47d6797b29306e, State: Initialized, Role: FOLLOWER
I20260806 16:03:58.968680 17152 consensus_queue.cc:260] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:58.969331 17123 heartbeater.cc:503] Master 127.13.148.62:39165 was elected leader, sending a full tablet report...
I20260806 16:03:58.969657 17152 ts_tablet_manager.cc:1428] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260806 16:03:59.121161 17124 tablet.cc:2369] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:03:59.121774 16913 tablet.cc:2369] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:03:59.139161 16716 tablet.cc:2369] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:03:59.307889 17159 raft_consensus.cc:482] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:03:59.308179 17159 raft_consensus.cc:504] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.309491 17159 leader_election.cc:288] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48f337453cdf494995511f074951c8fa (127.13.148.2:35117), 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:03:59.315160 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.310267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:55265 (local address 127.13.148.3:43467)
0806 16:03:59.310409 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:59.310416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:59.310431 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:03:59.310500 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:59.310503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:59.310546 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:59.310619 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:59.310949 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.311731 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.313812 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.314432 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.314593 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.314650 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:59.314948 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 16:03:59.315022 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:03:59.315160 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.309830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:35693 (local address 127.13.148.2:35117)
0806 16:03:59.309997 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:59.310004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:59.310145 (+ 141us) server_negotiation.cc:407] Connection header received
0806 16:03:59.310322 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:59.310325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:59.310378 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:59.310458 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:03:59.310920 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.311759 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.313812 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.314432 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.314621 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.314666 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:59.314948 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 16:03:59.315032 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:03:59.315263 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.309699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35117 (local address 127.13.148.1:35693)
0806 16:03:59.310024 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:59.310043 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:59.310186 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:59.310544 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:59.310548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:59.310568 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:59.310760 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.310771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.311906 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.311911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.313647 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.313661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.314613 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.314618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.314744 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.314824 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 16:03:59.314914 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:03:59.315022 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":160,"thread_start_us":139,"threads_started":1}
I20260806 16:03:59.315263 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.309845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:43467 (local address 127.13.148.1:55265)
0806 16:03:59.310190 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:59.310207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:59.310315 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:59.310649 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:59.310652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:59.310666 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:03:59.310825 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.310834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.311906 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.311910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.313679 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.313688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.314638 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.314642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.314747 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.314822 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 16:03:59.314914 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 16:03:59.315022 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":254,"thread_start_us":124,"threads_started":1}
I20260806 16:03:59.316140 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56fb8704ff644518fc8b87de80cf1b6" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61edb70fbb7a4716bb47d6797b29306e" is_pre_election: true
I20260806 16:03:59.316222 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56fb8704ff644518fc8b87de80cf1b6" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f337453cdf494995511f074951c8fa" is_pre_election: true
I20260806 16:03:59.316493 17051 raft_consensus.cc:2454] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 0.
I20260806 16:03:59.316535 16856 raft_consensus.cc:2454] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 0.
I20260806 16:03:59.317083 16558 leader_election.cc:302] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48f337453cdf494995511f074951c8fa, 61edb70fbb7a4716bb47d6797b29306e, b5456ab5278b40488b27912b3556de04; no voters:
I20260806 16:03:59.317394 17159 raft_consensus.cc:2781] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:03:59.317494 17159 raft_consensus.cc:482] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:03:59.317556 17159 raft_consensus.cc:3037] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.318646 17159 raft_consensus.cc:504] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.319065 17159 leader_election.cc:288] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 election: Requested vote from peers 48f337453cdf494995511f074951c8fa (127.13.148.2:35117), 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:03:59.319377 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56fb8704ff644518fc8b87de80cf1b6" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f337453cdf494995511f074951c8fa"
I20260806 16:03:59.319448 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56fb8704ff644518fc8b87de80cf1b6" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61edb70fbb7a4716bb47d6797b29306e"
I20260806 16:03:59.319520 16856 raft_consensus.cc:3037] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.319605 17051 raft_consensus.cc:3037] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.320549 16856 raft_consensus.cc:2454] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 1.
I20260806 16:03:59.320874 16560 leader_election.cc:302] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f337453cdf494995511f074951c8fa, b5456ab5278b40488b27912b3556de04; no voters:
I20260806 16:03:59.320879 17051 raft_consensus.cc:2454] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 1.
I20260806 16:03:59.321040 17159 raft_consensus.cc:2781] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:59.321269 17159 raft_consensus.cc:686] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 1 LEADER]: Becoming Leader. State: Replica: b5456ab5278b40488b27912b3556de04, State: Running, Role: LEADER
I20260806 16:03:59.321605 17159 consensus_queue.cc:237] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.325054 16439 catalog_manager.cc:5495] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5456ab5278b40488b27912b3556de04 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } health_report { overall_health: UNKNOWN } } }
Aug 06 16:03:59 dist-test-slave-7kzw krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032238, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:03:59.438186 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.428406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39165 (local address 127.0.0.1:49296)
0806 16:03:59.428744 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:59.428758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:59.428856 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:59.429136 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:59.429139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:59.429545 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:03:59.429643 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.429648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.430532 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.430539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.431109 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.431118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.431617 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.431621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.431692 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.432497 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:03:59.432519 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:03:59.434740 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:03:59.436447 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:59.436456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:59.436463 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:59.436726 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:59.437007 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:03:59.437010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:03:59.437011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:03:59.437100 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:03:59.437539 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:03:59.437545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:03:59.437673 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 16:03:59.437811 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 16:03:59.438009 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20260806 16:03:59.438359 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.428494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.13.148.62:39165)
0806 16:03:59.428831 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:59.428837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:59.428853 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:03:59.428934 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:59.428937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:59.428977 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:59.429055 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:03:59.429796 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.430343 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.431235 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.431479 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.431565 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.434880 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:03:59.434883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:03:59.434885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:03:59.434908 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:03:59.436336 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:59.436830 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:59.436833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:59.436835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:59.436918 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:03:59.437216 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:03:59.437219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:03:59.437220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:03:59.437406 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:03:59.437482 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:59.438027 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 16:03:59.438140 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":167,"threads_started":1}
I20260806 16:03:59.443092 16439 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:49296:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 16:03:59.443898 16439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:03:59.449610 16628 tablet_service.cc:1467] Processing CreateTablet for tablet d896cf7a4c734bd3b81c482701d44462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9bf29153773b4eee8f7f8b49250296d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:03:59.449769 16828 tablet_service.cc:1467] Processing CreateTablet for tablet d896cf7a4c734bd3b81c482701d44462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9bf29153773b4eee8f7f8b49250296d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:03:59.449748 17024 tablet_service.cc:1467] Processing CreateTablet for tablet d896cf7a4c734bd3b81c482701d44462 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9bf29153773b4eee8f7f8b49250296d9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:03:59.450405 16628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d896cf7a4c734bd3b81c482701d44462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:59.450538 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d896cf7a4c734bd3b81c482701d44462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:59.450542 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d896cf7a4c734bd3b81c482701d44462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:59.453095 17154 tablet_bootstrap.cc:492] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: Bootstrap starting.
I20260806 16:03:59.453737 17152 tablet_bootstrap.cc:492] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: Bootstrap starting.
I20260806 16:03:59.454130 17154 tablet_bootstrap.cc:654] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:59.454239 17151 tablet_bootstrap.cc:492] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: Bootstrap starting.
I20260806 16:03:59.454879 17152 tablet_bootstrap.cc:654] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:59.455327 17151 tablet_bootstrap.cc:654] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:59.455624 17154 tablet_bootstrap.cc:492] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: No bootstrap required, opened a new log
I20260806 16:03:59.455735 17154 ts_tablet_manager.cc:1397] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 16:03:59.456135 17154 raft_consensus.cc:348] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.456274 17154 raft_consensus.cc:374] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:59.456322 17154 raft_consensus.cc:729] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5456ab5278b40488b27912b3556de04, State: Initialized, Role: FOLLOWER
I20260806 16:03:59.456512 17154 consensus_queue.cc:260] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.456671 17152 tablet_bootstrap.cc:492] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: No bootstrap required, opened a new log
I20260806 16:03:59.456676 17151 tablet_bootstrap.cc:492] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: No bootstrap required, opened a new log
I20260806 16:03:59.456773 17152 ts_tablet_manager.cc:1397] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:03:59.456773 17151 ts_tablet_manager.cc:1397] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:03:59.456875 17154 ts_tablet_manager.cc:1428] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260806 16:03:59.457157 17159 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.457300 17159 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:59.457260 17151 raft_consensus.cc:348] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.457260 17152 raft_consensus.cc:348] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.457377 17151 raft_consensus.cc:374] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:59.457377 17152 raft_consensus.cc:374] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:59.457417 17151 raft_consensus.cc:729] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f337453cdf494995511f074951c8fa, State: Initialized, Role: FOLLOWER
I20260806 16:03:59.457417 17152 raft_consensus.cc:729] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61edb70fbb7a4716bb47d6797b29306e, State: Initialized, Role: FOLLOWER
I20260806 16:03:59.457567 17151 consensus_queue.cc:260] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.457630 17152 consensus_queue.cc:260] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.457872 17151 ts_tablet_manager.cc:1428] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:03:59.458137 17157 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.458328 17157 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:59.458352 17160 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.458518 17160 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:59.459609 17152 ts_tablet_manager.cc:1428] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260806 16:03:59.535161 17159 raft_consensus.cc:482] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:03:59.535312 17159 raft_consensus.cc:504] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.535682 17159 leader_election.cc:288] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48f337453cdf494995511f074951c8fa (127.13.148.2:35117), 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:03:59.536175 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d896cf7a4c734bd3b81c482701d44462" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f337453cdf494995511f074951c8fa" is_pre_election: true
I20260806 16:03:59.536257 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d896cf7a4c734bd3b81c482701d44462" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61edb70fbb7a4716bb47d6797b29306e" is_pre_election: true
I20260806 16:03:59.536345 16856 raft_consensus.cc:2454] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 0.
I20260806 16:03:59.536492 17051 raft_consensus.cc:2454] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 0.
I20260806 16:03:59.536685 16560 leader_election.cc:302] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f337453cdf494995511f074951c8fa, b5456ab5278b40488b27912b3556de04; no voters:
I20260806 16:03:59.536911 17159 raft_consensus.cc:2781] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:03:59.536991 17159 raft_consensus.cc:482] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:03:59.537055 17159 raft_consensus.cc:3037] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.538329 17159 raft_consensus.cc:504] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.538908 17159 leader_election.cc:288] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 election: Requested vote from peers 48f337453cdf494995511f074951c8fa (127.13.148.2:35117), 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:03:59.539379 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d896cf7a4c734bd3b81c482701d44462" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61edb70fbb7a4716bb47d6797b29306e"
I20260806 16:03:59.539362 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d896cf7a4c734bd3b81c482701d44462" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f337453cdf494995511f074951c8fa"
I20260806 16:03:59.539508 16856 raft_consensus.cc:3037] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.539508 17051 raft_consensus.cc:3037] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.540805 16856 raft_consensus.cc:2454] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 1.
I20260806 16:03:59.540805 17051 raft_consensus.cc:2454] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 1.
I20260806 16:03:59.541203 16558 leader_election.cc:302] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61edb70fbb7a4716bb47d6797b29306e, b5456ab5278b40488b27912b3556de04; no voters:
I20260806 16:03:59.541430 17159 raft_consensus.cc:2781] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:59.541564 17159 raft_consensus.cc:686] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [term 1 LEADER]: Becoming Leader. State: Replica: b5456ab5278b40488b27912b3556de04, State: Running, Role: LEADER
I20260806 16:03:59.541728 17159 consensus_queue.cc:237] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.542408 17164 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: TxnStatusTablet state changed. Reason: New leader b5456ab5278b40488b27912b3556de04. Latest consensus state: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.542557 17164 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:03:59.542896 17180 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 16:03:59.543180 17180 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 16:03:59.543604 16439 catalog_manager.cc:5495] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5456ab5278b40488b27912b3556de04 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } health_report { overall_health: UNKNOWN } } }
I20260806 16:03:59.565439 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.561078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.13.148.1:46595)
0806 16:03:59.561495 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:59.561503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:59.561516 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:03:59.561584 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:59.561586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:59.561650 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:59.561789 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:03:59.562098 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.562767 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.563727 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.564003 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.564129 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.564466 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:03:59.564925 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:03:59.565063 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:59.565175 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 16:03:59.565243 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":171,"threads_started":1}
I20260806 16:03:59.565415 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.560954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46595 (local address 127.0.0.1:34200)
0806 16:03:59.561156 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:59.561179 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:59.561326 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:59.561808 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:59.561811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:59.561820 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:03:59.561948 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.561954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.562895 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.562900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.563579 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.563597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.564134 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.564137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.564240 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.564289 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:03:59.565040 (+ 751us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:03:59.565061 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:03:59.565147 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 16:03:59.565244 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 16:03:59.571168 17051 raft_consensus.cc:1261] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Refusing update from remote peer b5456ab5278b40488b27912b3556de04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:59.571171 16856 raft_consensus.cc:1261] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Refusing update from remote peer b5456ab5278b40488b27912b3556de04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:59.571921 17164 consensus_queue.cc:1035] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:59.572145 17159 consensus_queue.cc:1035] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:59.576766 17160 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: TxnStatusTablet state changed. Reason: New leader b5456ab5278b40488b27912b3556de04. Latest consensus state: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.576907 17160 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:59.577355 17157 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: TxnStatusTablet state changed. Reason: New leader b5456ab5278b40488b27912b3556de04. Latest consensus state: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.577487 17157 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:59.581318 17159 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.581458 17159 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:03:59.581509 17164 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.581615 17164 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P b5456ab5278b40488b27912b3556de04: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:03:59.581805 17188 mvcc.cc:205] Tried to move back new op lower bound from 7315588053275570176 to 7315588053164281856. Current Snapshot: MvccSnapshot[applied={T|T < 7315588053275570176}]
I20260806 16:03:59.582382 17157 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.582516 17157 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P 48f337453cdf494995511f074951c8fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:59.582763 17160 tablet_replica.cc:439] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } } }
I20260806 16:03:59.582892 17160 tablet_replica.cc:442] T d896cf7a4c734bd3b81c482701d44462 P 61edb70fbb7a4716bb47d6797b29306e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:03:59.586036 17190 mvcc.cc:205] Tried to move back new op lower bound from 7315588053275570176 to 7315588053164281856. Current Snapshot: MvccSnapshot[applied={T|T < 7315588053275570176}]
I20260806 16:03:59.586575 17189 mvcc.cc:205] Tried to move back new op lower bound from 7315588053275570176 to 7315588053164281856. Current Snapshot: MvccSnapshot[applied={T|T < 7315588053275570176}]
I20260806 16:03:59.596685 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.592722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.13.148.1:46595)
0806 16:03:59.592875 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:59.592880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:59.593119 (+ 239us) server_negotiation.cc:407] Connection header received
0806 16:03:59.593279 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:59.593281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:59.593328 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:59.593434 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:03:59.593742 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.594388 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.595492 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.595751 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.595866 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.596198 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:03:59.596292 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:03:59.596373 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:59.596468 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 16:03:59.596527 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:03:59.609913 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.605342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:59899 (local address 127.13.148.1:46595)
0806 16:03:59.605473 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:03:59.605478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:03:59.605686 (+ 208us) server_negotiation.cc:407] Connection header received
0806 16:03:59.605881 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:03:59.605885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:03:59.605935 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:03:59.606023 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:03:59.606324 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.606961 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.608606 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.608860 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.608978 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.609340 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:03:59.609444 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:03:59.609532 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 16:03:59.609651 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 16:03:59.609729 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":32}
I20260806 16:03:59.609928 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:03:59.605146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46595 (local address 127.13.148.1:59899)
0806 16:03:59.605549 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 16:03:59.605575 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 16:03:59.605764 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:03:59.606032 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:03:59.606035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:03:59.606042 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:03:59.606203 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.606211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.607084 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.607088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.608472 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:03:59.608484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:03:59.608992 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:03:59.608995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:03:59.609127 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:03:59.609171 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:03:59.609546 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:03:59.609555 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 16:03:59.609639 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 16:03:59.609748 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":50,"thread_start_us":121,"threads_started":1}
I20260806 16:03:59.617503 16856 raft_consensus.cc:1261] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Refusing update from remote peer b5456ab5278b40488b27912b3556de04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:59.617667 17051 raft_consensus.cc:1261] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Refusing update from remote peer b5456ab5278b40488b27912b3556de04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:59.618108 17195 consensus_queue.cc:1035] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:59.618309 17159 consensus_queue.cc:1035] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:59.681351 16439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:03:59.681581 16439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:03:59.684727 16439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: Sending DeleteTablet for 3 replicas of tablet d56fb8704ff644518fc8b87de80cf1b6
I20260806 16:03:59.685730 16828 tablet_service.cc:1514] Processing DeleteTablet for tablet d56fb8704ff644518fc8b87de80cf1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:59 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:54516
I20260806 16:03:59.685748 17024 tablet_service.cc:1514] Processing DeleteTablet for tablet d56fb8704ff644518fc8b87de80cf1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:59 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:45556
I20260806 16:03:59.686354 17217 tablet_replica.cc:330] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: stopping tablet replica
I20260806 16:03:59.686367 16626 tablet_service.cc:1514] Processing DeleteTablet for tablet d56fb8704ff644518fc8b87de80cf1b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:03:59 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:34186
I20260806 16:03:59.686731 17216 tablet_replica.cc:330] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: stopping tablet replica
I20260806 16:03:59.686934 17218 tablet_replica.cc:330] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: stopping tablet replica
I20260806 16:03:59.687266 17218 raft_consensus.cc:2229] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:03:59.687217 16439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:03:59.687690 16439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:03:59.687681 17216 raft_consensus.cc:2229] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:03:59.687716 17218 raft_consensus.cc:2258] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:03:59.687322 17217 raft_consensus.cc:2229] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:03:59.688097 17217 raft_consensus.cc:2258] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:03:59.688459 17216 raft_consensus.cc:2258] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:03:59.690793 17216 ts_tablet_manager.cc:1905] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:59.693953 17217 ts_tablet_manager.cc:1905] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:59.693971 16626 tablet_service.cc:1467] Processing CreateTablet for tablet f2ca75f01beb47839294b05ca7cdadc5 (DEFAULT_TABLE table=test-table [id=14282b7c08204aceae277ed1dc3d4c9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:59.693974 17024 tablet_service.cc:1467] Processing CreateTablet for tablet f2ca75f01beb47839294b05ca7cdadc5 (DEFAULT_TABLE table=test-table [id=14282b7c08204aceae277ed1dc3d4c9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:59.694265 16626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ca75f01beb47839294b05ca7cdadc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:59.694281 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ca75f01beb47839294b05ca7cdadc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:59.697185 17152 tablet_bootstrap.cc:492] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: Bootstrap starting.
I20260806 16:03:59.697161 16828 tablet_service.cc:1467] Processing CreateTablet for tablet f2ca75f01beb47839294b05ca7cdadc5 (DEFAULT_TABLE table=test-table [id=14282b7c08204aceae277ed1dc3d4c9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:03:59.697238 17217 ts_tablet_manager.cc:1918] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:03:59.697335 17217 log.cc:1196] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/d56fb8704ff644518fc8b87de80cf1b6
I20260806 16:03:59.697439 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ca75f01beb47839294b05ca7cdadc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:03:59.697674 17217 ts_tablet_manager.cc:1939] T d56fb8704ff644518fc8b87de80cf1b6 P 48f337453cdf494995511f074951c8fa: Deleting consensus metadata
I20260806 16:03:59.698293 17152 tablet_bootstrap.cc:654] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:59.699105 16422 catalog_manager.cc:4841] TS 48f337453cdf494995511f074951c8fa (127.13.148.2:35117): tablet d56fb8704ff644518fc8b87de80cf1b6 (table test-table [id=88b67103b54f48c09e511293a5c633ef]) successfully deleted
I20260806 16:03:59.699535 17216 ts_tablet_manager.cc:1918] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:03:59.699638 17216 log.cc:1196] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/d56fb8704ff644518fc8b87de80cf1b6
I20260806 16:03:59.699962 17216 ts_tablet_manager.cc:1939] T d56fb8704ff644518fc8b87de80cf1b6 P 61edb70fbb7a4716bb47d6797b29306e: Deleting consensus metadata
I20260806 16:03:59.700650 17154 tablet_bootstrap.cc:492] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: Bootstrap starting.
I20260806 16:03:59.700788 17152 tablet_bootstrap.cc:492] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: No bootstrap required, opened a new log
I20260806 16:03:59.700873 17152 ts_tablet_manager.cc:1397] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260806 16:03:59.701428 16420 catalog_manager.cc:4841] TS 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467): tablet d56fb8704ff644518fc8b87de80cf1b6 (table test-table [id=88b67103b54f48c09e511293a5c633ef]) successfully deleted
I20260806 16:03:59.701586 17154 tablet_bootstrap.cc:654] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:59.702188 17152 raft_consensus.cc:348] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.702314 17152 raft_consensus.cc:374] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:59.702347 17152 raft_consensus.cc:729] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61edb70fbb7a4716bb47d6797b29306e, State: Initialized, Role: FOLLOWER
I20260806 16:03:59.702517 17152 consensus_queue.cc:260] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.702802 17152 ts_tablet_manager.cc:1428] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 16:03:59.702917 17154 tablet_bootstrap.cc:492] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: No bootstrap required, opened a new log
I20260806 16:03:59.702991 17154 ts_tablet_manager.cc:1397] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 16:03:59.703336 17154 raft_consensus.cc:348] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.703416 17154 raft_consensus.cc:374] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:59.703438 17154 raft_consensus.cc:729] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5456ab5278b40488b27912b3556de04, State: Initialized, Role: FOLLOWER
I20260806 16:03:59.703572 17154 consensus_queue.cc:260] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.703845 17154 ts_tablet_manager.cc:1428] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:03:59.704094 17151 tablet_bootstrap.cc:492] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: Bootstrap starting.
I20260806 16:03:59.705144 17151 tablet_bootstrap.cc:654] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: Neither blocks nor log segments found. Creating new log.
I20260806 16:03:59.706198 17218 ts_tablet_manager.cc:1905] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:03:59.707326 17151 tablet_bootstrap.cc:492] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: No bootstrap required, opened a new log
I20260806 16:03:59.707422 17151 ts_tablet_manager.cc:1397] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:03:59.707864 17151 raft_consensus.cc:348] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.707975 17151 raft_consensus.cc:374] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:03:59.708007 17151 raft_consensus.cc:729] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f337453cdf494995511f074951c8fa, State: Initialized, Role: FOLLOWER
I20260806 16:03:59.708140 17151 consensus_queue.cc:260] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.708590 17151 ts_tablet_manager.cc:1428] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:03:59.709987 17218 ts_tablet_manager.cc:1918] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:03:59.710129 17218 log.cc:1196] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/d56fb8704ff644518fc8b87de80cf1b6
I20260806 16:03:59.710567 17218 ts_tablet_manager.cc:1939] T d56fb8704ff644518fc8b87de80cf1b6 P b5456ab5278b40488b27912b3556de04: Deleting consensus metadata
I20260806 16:03:59.711807 16420 catalog_manager.cc:4841] TS b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595): tablet d56fb8704ff644518fc8b87de80cf1b6 (table test-table [id=88b67103b54f48c09e511293a5c633ef]) successfully deleted
I20260806 16:03:59.780740 17164 raft_consensus.cc:482] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:03:59.780949 17164 raft_consensus.cc:504] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.781443 17164 leader_election.cc:288] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48f337453cdf494995511f074951c8fa (127.13.148.2:35117), 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:03:59.781944 16855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ca75f01beb47839294b05ca7cdadc5" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f337453cdf494995511f074951c8fa" is_pre_election: true
I20260806 16:03:59.781942 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ca75f01beb47839294b05ca7cdadc5" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61edb70fbb7a4716bb47d6797b29306e" is_pre_election: true
I20260806 16:03:59.782101 17051 raft_consensus.cc:2454] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 0.
I20260806 16:03:59.782101 16855 raft_consensus.cc:2454] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 0.
I20260806 16:03:59.782441 16558 leader_election.cc:302] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48f337453cdf494995511f074951c8fa, 61edb70fbb7a4716bb47d6797b29306e, b5456ab5278b40488b27912b3556de04; no voters:
I20260806 16:03:59.782648 17164 raft_consensus.cc:2781] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:03:59.782714 17164 raft_consensus.cc:482] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:03:59.782769 17164 raft_consensus.cc:3037] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.783804 17164 raft_consensus.cc:504] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.784191 17164 leader_election.cc:288] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 election: Requested vote from peers 48f337453cdf494995511f074951c8fa (127.13.148.2:35117), 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:03:59.784543 16855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ca75f01beb47839294b05ca7cdadc5" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f337453cdf494995511f074951c8fa"
I20260806 16:03:59.784622 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ca75f01beb47839294b05ca7cdadc5" candidate_uuid: "b5456ab5278b40488b27912b3556de04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61edb70fbb7a4716bb47d6797b29306e"
I20260806 16:03:59.784695 16855 raft_consensus.cc:3037] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.784780 17051 raft_consensus.cc:3037] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:03:59.785709 16855 raft_consensus.cc:2454] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 1.
I20260806 16:03:59.785943 17051 raft_consensus.cc:2454] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5456ab5278b40488b27912b3556de04 in term 1.
I20260806 16:03:59.786060 16560 leader_election.cc:302] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f337453cdf494995511f074951c8fa, b5456ab5278b40488b27912b3556de04; no voters:
I20260806 16:03:59.786271 17164 raft_consensus.cc:2781] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:03:59.786368 17164 raft_consensus.cc:686] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 1 LEADER]: Becoming Leader. State: Replica: b5456ab5278b40488b27912b3556de04, State: Running, Role: LEADER
I20260806 16:03:59.786556 17164 consensus_queue.cc:237] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:03:59.788430 16437 catalog_manager.cc:5495] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5456ab5278b40488b27912b3556de04 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "b5456ab5278b40488b27912b3556de04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } health_report { overall_health: UNKNOWN } } }
I20260806 16:03:59.823262 17051 raft_consensus.cc:1261] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Refusing update from remote peer b5456ab5278b40488b27912b3556de04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:59.823523 16855 raft_consensus.cc:1261] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Refusing update from remote peer b5456ab5278b40488b27912b3556de04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:03:59.823737 17164 consensus_queue.cc:1035] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:03:59.823940 17212 consensus_queue.cc:1035] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 16:03:59.841563 17188 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 16:04:01.400372 16439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:01.400550 16439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:01.403160 16439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: Sending DeleteTablet for 3 replicas of tablet f2ca75f01beb47839294b05ca7cdadc5
I20260806 16:04:01.403828 16828 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ca75f01beb47839294b05ca7cdadc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:54516
I20260806 16:04:01.403949 17024 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ca75f01beb47839294b05ca7cdadc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:45556
I20260806 16:04:01.404734 16626 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ca75f01beb47839294b05ca7cdadc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:34186
I20260806 16:04:01.405102 17263 tablet_replica.cc:330] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: stopping tablet replica
I20260806 16:04:01.405212 17261 tablet_replica.cc:330] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: stopping tablet replica
I20260806 16:04:01.405282 17263 raft_consensus.cc:2229] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:01.405361 17261 raft_consensus.cc:2229] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:01.405498 17261 raft_consensus.cc:2258] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:01.405499 17263 raft_consensus.cc:2258] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:01.405793 17259 tablet_replica.cc:330] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: stopping tablet replica
I20260806 16:04:01.405933 17259 raft_consensus.cc:2229] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:01.406605 17261 ts_tablet_manager.cc:1905] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:01.407169 17263 ts_tablet_manager.cc:1905] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:01.409813 17261 ts_tablet_manager.cc:1918] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:04:01.409926 17261 log.cc:1196] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/f2ca75f01beb47839294b05ca7cdadc5
I20260806 16:04:01.410116 17259 raft_consensus.cc:2258] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:01.410372 17261 ts_tablet_manager.cc:1939] T f2ca75f01beb47839294b05ca7cdadc5 P 61edb70fbb7a4716bb47d6797b29306e: Deleting consensus metadata
I20260806 16:04:01.410756 17263 ts_tablet_manager.cc:1918] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:04:01.410880 17263 log.cc:1196] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/f2ca75f01beb47839294b05ca7cdadc5
I20260806 16:04:01.410976 17259 ts_tablet_manager.cc:1905] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:01.411273 17263 ts_tablet_manager.cc:1939] T f2ca75f01beb47839294b05ca7cdadc5 P b5456ab5278b40488b27912b3556de04: Deleting consensus metadata
I20260806 16:04:01.411510 16420 catalog_manager.cc:4841] TS 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467): tablet f2ca75f01beb47839294b05ca7cdadc5 (table test-table [id=14282b7c08204aceae277ed1dc3d4c9c]) successfully deleted
I20260806 16:04:01.412086 16420 catalog_manager.cc:4841] TS b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595): tablet f2ca75f01beb47839294b05ca7cdadc5 (table test-table [id=14282b7c08204aceae277ed1dc3d4c9c]) successfully deleted
I20260806 16:04:01.413666 17259 ts_tablet_manager.cc:1918] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:04:01.413772 17259 log.cc:1196] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/f2ca75f01beb47839294b05ca7cdadc5
I20260806 16:04:01.414108 17259 ts_tablet_manager.cc:1939] T f2ca75f01beb47839294b05ca7cdadc5 P 48f337453cdf494995511f074951c8fa: Deleting consensus metadata
I20260806 16:04:01.414943 16422 catalog_manager.cc:4841] TS 48f337453cdf494995511f074951c8fa (127.13.148.2:35117): tablet f2ca75f01beb47839294b05ca7cdadc5 (table test-table [id=14282b7c08204aceae277ed1dc3d4c9c]) successfully deleted
Aug 06 16:04:01 dist-test-slave-7kzw krb5kdc[16381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:04:01 dist-test-slave-7kzw krb5kdc[16381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:01.436111 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.426567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.13.148.62:39165)
0806 16:04:01.426949 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.426960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.427043 (+ 83us) server_negotiation.cc:407] Connection header received
0806 16:04:01.427250 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.427254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.427326 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.427427 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:01.428199 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.428771 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.429640 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.429887 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.429997 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.432741 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:01.432745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:01.432747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:01.432785 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:01.434253 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:01.434768 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:01.434772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:01.434774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:01.434869 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:01.435181 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:01.435184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:01.435186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:01.435382 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:01.435528 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.435787 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.435925 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260806 16:04:01.439029 16439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:01.439587 16439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:01.446077 16626 tablet_service.cc:1467] Processing CreateTablet for tablet 52a6488a7d944550babf14c66a8fa44b (DEFAULT_TABLE table=test-table [id=2f9d77c679494f96ae6fffe6b8602d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:01.446141 16828 tablet_service.cc:1467] Processing CreateTablet for tablet 52a6488a7d944550babf14c66a8fa44b (DEFAULT_TABLE table=test-table [id=2f9d77c679494f96ae6fffe6b8602d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:01.446274 17024 tablet_service.cc:1467] Processing CreateTablet for tablet 52a6488a7d944550babf14c66a8fa44b (DEFAULT_TABLE table=test-table [id=2f9d77c679494f96ae6fffe6b8602d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:01.446434 16626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a6488a7d944550babf14c66a8fa44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:01.446458 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a6488a7d944550babf14c66a8fa44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:01.446579 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a6488a7d944550babf14c66a8fa44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:01.448612 17275 tablet_bootstrap.cc:492] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: Bootstrap starting.
I20260806 16:04:01.449488 17278 tablet_bootstrap.cc:492] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: Bootstrap starting.
I20260806 16:04:01.449535 17275 tablet_bootstrap.cc:654] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:01.449735 17279 tablet_bootstrap.cc:492] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: Bootstrap starting.
I20260806 16:04:01.450425 17278 tablet_bootstrap.cc:654] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:01.450721 17279 tablet_bootstrap.cc:654] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:01.451874 17279 tablet_bootstrap.cc:492] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: No bootstrap required, opened a new log
I20260806 16:04:01.451954 17279 ts_tablet_manager.cc:1397] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:04:01.452093 17278 tablet_bootstrap.cc:492] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: No bootstrap required, opened a new log
I20260806 16:04:01.452183 17278 ts_tablet_manager.cc:1397] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 16:04:01.452443 17279 raft_consensus.cc:348] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.452561 17279 raft_consensus.cc:374] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:01.452595 17279 raft_consensus.cc:729] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5456ab5278b40488b27912b3556de04, State: Initialized, Role: FOLLOWER
I20260806 16:04:01.452745 17279 consensus_queue.cc:260] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.452802 17278 raft_consensus.cc:348] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.452920 17278 raft_consensus.cc:374] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:01.452960 17278 raft_consensus.cc:729] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61edb70fbb7a4716bb47d6797b29306e, State: Initialized, Role: FOLLOWER
I20260806 16:04:01.453069 17279 ts_tablet_manager.cc:1428] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:04:01.453087 17278 consensus_queue.cc:260] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.453488 17278 ts_tablet_manager.cc:1428] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260806 16:04:01.453683 17275 tablet_bootstrap.cc:492] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: No bootstrap required, opened a new log
I20260806 16:04:01.453786 17275 ts_tablet_manager.cc:1397] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 16:04:01.454213 17275 raft_consensus.cc:348] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.454332 17275 raft_consensus.cc:374] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:01.454365 17275 raft_consensus.cc:729] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f337453cdf494995511f074951c8fa, State: Initialized, Role: FOLLOWER
I20260806 16:04:01.454588 17275 consensus_queue.cc:260] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.455054 17275 ts_tablet_manager.cc:1428] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 16:04:01.533272 17291 raft_consensus.cc:482] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:01.533447 17291 raft_consensus.cc:504] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.534783 17291 leader_election.cc:288] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595), 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:04:01.540536 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.535233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:57451 (local address 127.13.148.1:46595)
0806 16:04:01.535578 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.535635 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.535655 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:04:01.535735 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.535738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.535797 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.535918 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:01.536307 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.537236 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.539135 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.539788 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.539924 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.539980 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.540310 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.540390 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":154,"threads_started":1}
I20260806 16:04:01.540550 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.535738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:34813 (local address 127.13.148.3:43467)
0806 16:04:01.536071 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.536079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.536099 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:04:01.536146 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.536148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.536187 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.536307 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:01.536572 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.537414 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.539180 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.539797 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.539924 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.539980 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.540310 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.540389 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
I20260806 16:04:01.540570 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.535085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46595 (local address 127.13.148.2:57451)
0806 16:04:01.535431 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:01.535451 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:01.535565 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:01.535950 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:01.535954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:01.535999 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:01.536164 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.536170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.537376 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.537378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.538986 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.538998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.539941 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.539946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.540127 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.540184 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 16:04:01.540265 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 16:04:01.540385 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":132,"spinlock_wait_cycles":28288,"thread_start_us":169,"threads_started":1}
I20260806 16:04:01.540580 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.535559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:43467 (local address 127.13.148.2:34813)
0806 16:04:01.535853 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:01.535866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:01.536006 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:01.536330 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:01.536332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:01.536349 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:01.536449 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.536454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.537539 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.537542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.538988 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.538998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.539941 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.539946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.540149 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.540181 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 16:04:01.540265 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 16:04:01.540381 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":164,"spinlock_wait_cycles":8320,"thread_start_us":125,"threads_started":1}
I20260806 16:04:01.541159 17050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a6488a7d944550babf14c66a8fa44b" candidate_uuid: "48f337453cdf494995511f074951c8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61edb70fbb7a4716bb47d6797b29306e" is_pre_election: true
I20260806 16:04:01.541328 17050 raft_consensus.cc:2454] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f337453cdf494995511f074951c8fa in term 0.
I20260806 16:04:01.541570 16652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a6488a7d944550babf14c66a8fa44b" candidate_uuid: "48f337453cdf494995511f074951c8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5456ab5278b40488b27912b3556de04" is_pre_election: true
I20260806 16:04:01.541783 16750 leader_election.cc:302] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f337453cdf494995511f074951c8fa, 61edb70fbb7a4716bb47d6797b29306e; no voters:
I20260806 16:04:01.541842 16652 raft_consensus.cc:2454] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f337453cdf494995511f074951c8fa in term 0.
I20260806 16:04:01.542085 17291 raft_consensus.cc:2781] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:01.542184 17291 raft_consensus.cc:482] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:01.542243 17291 raft_consensus.cc:3037] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:01.543301 17291 raft_consensus.cc:504] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.543668 17291 leader_election.cc:288] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [CANDIDATE]: Term 1 election: Requested vote from peers b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595), 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467)
I20260806 16:04:01.544111 16652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a6488a7d944550babf14c66a8fa44b" candidate_uuid: "48f337453cdf494995511f074951c8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5456ab5278b40488b27912b3556de04"
I20260806 16:04:01.544234 16652 raft_consensus.cc:3037] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:01.544276 17050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a6488a7d944550babf14c66a8fa44b" candidate_uuid: "48f337453cdf494995511f074951c8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61edb70fbb7a4716bb47d6797b29306e"
I20260806 16:04:01.544432 17050 raft_consensus.cc:3037] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:01.545315 16652 raft_consensus.cc:2454] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f337453cdf494995511f074951c8fa in term 1.
I20260806 16:04:01.545528 17050 raft_consensus.cc:2454] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f337453cdf494995511f074951c8fa in term 1.
I20260806 16:04:01.545627 16750 leader_election.cc:302] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f337453cdf494995511f074951c8fa, b5456ab5278b40488b27912b3556de04; no voters:
I20260806 16:04:01.545845 17291 raft_consensus.cc:2781] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:01.546100 17291 raft_consensus.cc:686] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 1 LEADER]: Becoming Leader. State: Replica: 48f337453cdf494995511f074951c8fa, State: Running, Role: LEADER
I20260806 16:04:01.546372 17291 consensus_queue.cc:237] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.548985 16434 catalog_manager.cc:5495] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 48f337453cdf494995511f074951c8fa (127.13.148.2). New cstate: current_term: 1 leader_uuid: "48f337453cdf494995511f074951c8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } health_report { overall_health: UNKNOWN } } }
I20260806 16:04:01.589668 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.585656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.13.148.2:35117)
0806 16:04:01.585979 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.585986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.585997 (+ 11us) server_negotiation.cc:407] Connection header received
0806 16:04:01.586079 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.586081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.586137 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.586285 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:01.586572 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.587119 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.588000 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.588241 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.588352 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.588705 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.589212 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.589341 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.589401 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.589458 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":119,"threads_started":1}
I20260806 16:04:01.600838 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.597542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:44135 (local address 127.13.148.1:46595)
0806 16:04:01.597746 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.597751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.597871 (+ 120us) server_negotiation.cc:407] Connection header received
0806 16:04:01.597972 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.597974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.598021 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.598086 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:01.598330 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.599019 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.599898 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.600108 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.600195 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.600469 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.600534 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.600588 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.600682 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.600725 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 16:04:01.600858 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.597432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46595 (local address 127.13.148.2:44135)
0806 16:04:01.597787 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:01.597802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:01.597899 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:01.598111 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:01.598113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:01.598119 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:04:01.598236 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.598241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.599152 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.599154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.599778 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.599787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.600224 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.600227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.600302 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.600347 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.600614 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.600618 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:04:01.600676 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 16:04:01.600736 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":132,"threads_started":1}
I20260806 16:04:01.607401 17050 raft_consensus.cc:1261] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Refusing update from remote peer 48f337453cdf494995511f074951c8fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:01.607749 16652 raft_consensus.cc:1261] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Refusing update from remote peer 48f337453cdf494995511f074951c8fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:01.608043 17298 consensus_queue.cc:1035] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:01.608311 17291 consensus_queue.cc:1035] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:01.628142 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.623969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35117 (local address 127.13.148.1:55743)
0806 16:04:01.624324 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:01.624338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:01.624500 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:01.624845 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:01.624848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:01.624856 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:04:01.624979 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.624986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.626091 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.626094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.626889 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.626899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.627485 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.627488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.627554 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.627582 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.627900 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.627905 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:04:01.627954 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 16:04:01.628018 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260806 16:04:01.628141 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.624055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:55743 (local address 127.13.148.2:35117)
0806 16:04:01.624217 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.624221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.624466 (+ 245us) server_negotiation.cc:407] Connection header received
0806 16:04:01.624670 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.624674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.624733 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.624832 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:01.625105 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.625975 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.627020 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.627369 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.627471 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.627733 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.627819 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.627888 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.627966 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.628018 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 16:04:01.652963 16434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:01.653239 16434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:01.655792 16434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: Sending DeleteTablet for 3 replicas of tablet 52a6488a7d944550babf14c66a8fa44b
I20260806 16:04:01.656438 16828 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a6488a7d944550babf14c66a8fa44b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:54516
I20260806 16:04:01.656623 17259 tablet_replica.cc:330] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: stopping tablet replica
I20260806 16:04:01.656740 17259 raft_consensus.cc:2229] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:01.656708 17024 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a6488a7d944550babf14c66a8fa44b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:45556
I20260806 16:04:01.656922 17261 tablet_replica.cc:330] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: stopping tablet replica
I20260806 16:04:01.657042 17261 raft_consensus.cc:2229] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:01.657054 17259 raft_consensus.cc:2258] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:01.657161 17261 raft_consensus.cc:2258] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:01.657171 16434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 16:04:01.657469 16626 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a6488a7d944550babf14c66a8fa44b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:34186
W20260806 16:04:01.657591 16434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:01.657712 17263 tablet_replica.cc:330] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: stopping tablet replica
I20260806 16:04:01.657838 17263 raft_consensus.cc:2229] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:01.657951 17263 raft_consensus.cc:2258] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:01.658288 17261 ts_tablet_manager.cc:1905] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:01.658941 17263 ts_tablet_manager.cc:1905] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:01.661590 17263 ts_tablet_manager.cc:1918] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:01.662099 17259 ts_tablet_manager.cc:1905] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:01.664144 16828 tablet_service.cc:1467] Processing CreateTablet for tablet fcae30d832414116a73d903f3877e54f (DEFAULT_TABLE table=test-table [id=fcba2d23d3ff4e349a20054159ef63c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:01.664412 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae30d832414116a73d903f3877e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:01.664538 17259 ts_tablet_manager.cc:1918] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:01.664597 17259 log.cc:1196] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/52a6488a7d944550babf14c66a8fa44b
I20260806 16:04:01.664863 17259 ts_tablet_manager.cc:1939] T 52a6488a7d944550babf14c66a8fa44b P 48f337453cdf494995511f074951c8fa: Deleting consensus metadata
I20260806 16:04:01.665509 16422 catalog_manager.cc:4841] TS 48f337453cdf494995511f074951c8fa (127.13.148.2:35117): tablet 52a6488a7d944550babf14c66a8fa44b (table test-table [id=2f9d77c679494f96ae6fffe6b8602d24]) successfully deleted
I20260806 16:04:01.666320 17263 log.cc:1196] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/52a6488a7d944550babf14c66a8fa44b
I20260806 16:04:01.666723 17263 ts_tablet_manager.cc:1939] T 52a6488a7d944550babf14c66a8fa44b P b5456ab5278b40488b27912b3556de04: Deleting consensus metadata
I20260806 16:04:01.666946 17275 tablet_bootstrap.cc:492] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: Bootstrap starting.
I20260806 16:04:01.668249 16420 catalog_manager.cc:4841] TS b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595): tablet 52a6488a7d944550babf14c66a8fa44b (table test-table [id=2f9d77c679494f96ae6fffe6b8602d24]) successfully deleted
I20260806 16:04:01.668613 17261 ts_tablet_manager.cc:1918] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:01.668699 17261 log.cc:1196] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/52a6488a7d944550babf14c66a8fa44b
I20260806 16:04:01.668807 17024 tablet_service.cc:1467] Processing CreateTablet for tablet fcae30d832414116a73d903f3877e54f (DEFAULT_TABLE table=test-table [id=fcba2d23d3ff4e349a20054159ef63c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:01.668970 17261 ts_tablet_manager.cc:1939] T 52a6488a7d944550babf14c66a8fa44b P 61edb70fbb7a4716bb47d6797b29306e: Deleting consensus metadata
I20260806 16:04:01.669102 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae30d832414116a73d903f3877e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:01.669265 17275 tablet_bootstrap.cc:654] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:01.670120 16420 catalog_manager.cc:4841] TS 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467): tablet 52a6488a7d944550babf14c66a8fa44b (table test-table [id=2f9d77c679494f96ae6fffe6b8602d24]) successfully deleted
I20260806 16:04:01.670591 16626 tablet_service.cc:1467] Processing CreateTablet for tablet fcae30d832414116a73d903f3877e54f (DEFAULT_TABLE table=test-table [id=fcba2d23d3ff4e349a20054159ef63c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:01.670866 16626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae30d832414116a73d903f3877e54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:01.671401 17275 tablet_bootstrap.cc:492] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: No bootstrap required, opened a new log
I20260806 16:04:01.671473 17275 ts_tablet_manager.cc:1397] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 16:04:01.671555 17278 tablet_bootstrap.cc:492] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: Bootstrap starting.
I20260806 16:04:01.671900 17275 raft_consensus.cc:348] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.672015 17275 raft_consensus.cc:374] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:01.672056 17275 raft_consensus.cc:729] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f337453cdf494995511f074951c8fa, State: Initialized, Role: FOLLOWER
I20260806 16:04:01.672200 17275 consensus_queue.cc:260] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.672488 17275 ts_tablet_manager.cc:1428] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:01.672547 17278 tablet_bootstrap.cc:654] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:01.673862 17279 tablet_bootstrap.cc:492] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04: Bootstrap starting.
I20260806 16:04:01.673892 17278 tablet_bootstrap.cc:492] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: No bootstrap required, opened a new log
I20260806 16:04:01.673977 17278 ts_tablet_manager.cc:1397] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 16:04:01.674451 17278 raft_consensus.cc:348] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.674594 17278 raft_consensus.cc:374] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:01.674628 17278 raft_consensus.cc:729] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61edb70fbb7a4716bb47d6797b29306e, State: Initialized, Role: FOLLOWER
I20260806 16:04:01.674770 17279 tablet_bootstrap.cc:654] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:01.674753 17278 consensus_queue.cc:260] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.675009 17278 ts_tablet_manager.cc:1428] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 16:04:01.676079 17279 tablet_bootstrap.cc:492] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04: No bootstrap required, opened a new log
I20260806 16:04:01.676167 17279 ts_tablet_manager.cc:1397] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:04:01.676637 17279 raft_consensus.cc:348] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.676764 17279 raft_consensus.cc:374] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:01.676802 17279 raft_consensus.cc:729] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5456ab5278b40488b27912b3556de04, State: Initialized, Role: FOLLOWER
I20260806 16:04:01.676926 17279 consensus_queue.cc:260] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.677265 17279 ts_tablet_manager.cc:1428] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:01.718570 17284 raft_consensus.cc:482] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:01.718747 17284 raft_consensus.cc:504] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.720005 17284 leader_election.cc:288] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595), 48f337453cdf494995511f074951c8fa (127.13.148.2:35117)
I20260806 16:04:01.725167 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.720378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50871 (local address 127.13.148.1:46595)
0806 16:04:01.720501 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.720506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.720838 (+ 332us) server_negotiation.cc:407] Connection header received
0806 16:04:01.721055 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.721058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.721107 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.721180 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:01.721485 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.721982 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.723701 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.724347 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.724472 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.724530 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.724903 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.724983 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:04:01.725206 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.720234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46595 (local address 127.13.148.3:50871)
0806 16:04:01.720708 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:01.720732 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:01.720919 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:01.721219 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:01.721222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:01.721237 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:01.721371 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.721384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.722099 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.722104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.723569 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.723581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.724467 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.724471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.724634 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.724689 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 16:04:01.724884 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 16:04:01.724999 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":290,"thread_start_us":100,"threads_started":1}
I20260806 16:04:01.725697 16652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae30d832414116a73d903f3877e54f" candidate_uuid: "61edb70fbb7a4716bb47d6797b29306e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5456ab5278b40488b27912b3556de04" is_pre_election: true
I20260806 16:04:01.725890 16652 raft_consensus.cc:2454] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61edb70fbb7a4716bb47d6797b29306e in term 0.
I20260806 16:04:01.726313 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.720418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:57835 (local address 127.13.148.2:35117)
0806 16:04:01.720553 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.720560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.720803 (+ 243us) server_negotiation.cc:407] Connection header received
0806 16:04:01.720914 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.720917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.720976 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.721058 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:01.721374 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.722183 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.724592 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.725197 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.725319 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.725369 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.726082 (+ 713us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.726176 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:04:01.726294 16952 leader_election.cc:302] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61edb70fbb7a4716bb47d6797b29306e, b5456ab5278b40488b27912b3556de04; no voters:
I20260806 16:04:01.726696 17284 raft_consensus.cc:2781] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:01.726718 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.720234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35117 (local address 127.13.148.3:57835)
0806 16:04:01.720683 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:01.720698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:01.720819 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:01.721074 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:01.721077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:01.721091 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:01.721248 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.721256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.722317 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.722320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.723569 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.723581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.725815 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.725819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.725932 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.725957 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:04:01.726491 (+ 534us) client_negotiation.cc:240] Negotiation successful
0806 16:04:01.726572 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":57,"thread_start_us":92,"threads_started":1}
I20260806 16:04:01.726790 17284 raft_consensus.cc:482] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:01.726846 17284 raft_consensus.cc:3037] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:01.727303 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae30d832414116a73d903f3877e54f" candidate_uuid: "61edb70fbb7a4716bb47d6797b29306e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f337453cdf494995511f074951c8fa" is_pre_election: true
I20260806 16:04:01.727451 16856 raft_consensus.cc:2454] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61edb70fbb7a4716bb47d6797b29306e in term 0.
I20260806 16:04:01.728251 17284 raft_consensus.cc:504] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.728652 17284 leader_election.cc:288] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [CANDIDATE]: Term 1 election: Requested vote from peers b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595), 48f337453cdf494995511f074951c8fa (127.13.148.2:35117)
I20260806 16:04:01.728955 16652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae30d832414116a73d903f3877e54f" candidate_uuid: "61edb70fbb7a4716bb47d6797b29306e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5456ab5278b40488b27912b3556de04"
I20260806 16:04:01.729023 16856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae30d832414116a73d903f3877e54f" candidate_uuid: "61edb70fbb7a4716bb47d6797b29306e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f337453cdf494995511f074951c8fa"
I20260806 16:04:01.729086 16652 raft_consensus.cc:3037] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:01.729126 16856 raft_consensus.cc:3037] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:01.730254 16652 raft_consensus.cc:2454] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61edb70fbb7a4716bb47d6797b29306e in term 1.
I20260806 16:04:01.730254 16856 raft_consensus.cc:2454] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61edb70fbb7a4716bb47d6797b29306e in term 1.
I20260806 16:04:01.730716 16954 leader_election.cc:302] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f337453cdf494995511f074951c8fa, 61edb70fbb7a4716bb47d6797b29306e; no voters:
I20260806 16:04:01.730990 17284 raft_consensus.cc:2781] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:01.731242 17284 raft_consensus.cc:686] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 1 LEADER]: Becoming Leader. State: Replica: 61edb70fbb7a4716bb47d6797b29306e, State: Running, Role: LEADER
I20260806 16:04:01.731505 17284 consensus_queue.cc:237] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } }
I20260806 16:04:01.733875 16440 catalog_manager.cc:5495] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e reported cstate change: term changed from 0 to 1, leader changed from <none> to 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3). New cstate: current_term: 1 leader_uuid: "61edb70fbb7a4716bb47d6797b29306e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61edb70fbb7a4716bb47d6797b29306e" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43467 } health_report { overall_health: HEALTHY } } }
I20260806 16:04:01.761606 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.756921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.13.148.3:43467)
0806 16:04:01.757070 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.757076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.757139 (+ 63us) server_negotiation.cc:407] Connection header received
0806 16:04:01.757360 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.757364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.757451 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.757588 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:01.757883 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.758782 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.759704 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.760114 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.760234 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.760535 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.761162 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.761304 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.761346 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.761415 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:04:01.772336 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.769043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:34659 (local address 127.13.148.1:46595)
0806 16:04:01.769213 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.769218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.769233 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:04:01.769366 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.769369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.769421 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.769504 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:01.769792 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.770299 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.771251 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.771469 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.771578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.771888 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.771959 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.772035 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.772128 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.772187 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 16:04:01.772356 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.768780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46595 (local address 127.13.148.3:34659)
0806 16:04:01.769123 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:01.769140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:01.769243 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:01.769519 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:01.769522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:01.769530 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:04:01.769676 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.769682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.770417 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.770422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.771117 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.771126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.771582 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.771585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.771695 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.771738 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.772040 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.772049 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 16:04:01.772118 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 16:04:01.772201 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
I20260806 16:04:01.779556 16652 raft_consensus.cc:1261] T fcae30d832414116a73d903f3877e54f P b5456ab5278b40488b27912b3556de04 [term 1 FOLLOWER]: Refusing update from remote peer 61edb70fbb7a4716bb47d6797b29306e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:01.779987 16856 raft_consensus.cc:1261] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Refusing update from remote peer 61edb70fbb7a4716bb47d6797b29306e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:01.780179 17320 consensus_queue.cc:1035] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5456ab5278b40488b27912b3556de04" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:01.780364 17284 consensus_queue.cc:1035] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f337453cdf494995511f074951c8fa" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:01.800397 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.796241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:43467 (local address 127.13.148.1:34419)
0806 16:04:01.796411 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:01.796428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:01.796509 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:01.796983 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:01.796986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:01.796995 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:04:01.797107 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.797115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.798276 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.798279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.799029 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:01.799044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.799628 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.799632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:01.799725 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.799762 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.800106 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.800114 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:04:01.800198 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 16:04:01.800284 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 16:04:01.800439 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:01.796442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:34419 (local address 127.13.148.3:43467)
0806 16:04:01.796675 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:01.796679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:01.796692 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:04:01.796737 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:01.796740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:01.796789 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:01.796862 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:01.797375 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.798154 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.799202 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:01.799504 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:01.799628 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:01.799911 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:01.800001 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:01.800088 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:01.800247 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 16:04:01.800303 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
W20260806 16:04:01.801342 17236 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 16:04:03.370325 16440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:03.370504 16440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:03.373109 16440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2aaaf5400dd4993bef0eaaacac48c99: Sending DeleteTablet for 3 replicas of tablet fcae30d832414116a73d903f3877e54f
I20260806 16:04:03.373914 17024 tablet_service.cc:1514] Processing DeleteTablet for tablet fcae30d832414116a73d903f3877e54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:45556
I20260806 16:04:03.374001 16626 tablet_service.cc:1514] Processing DeleteTablet for tablet fcae30d832414116a73d903f3877e54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:34186
I20260806 16:04:03.374305 16828 tablet_service.cc:1514] Processing DeleteTablet for tablet fcae30d832414116a73d903f3877e54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:54516
I20260806 16:04:03.374348 17374 tablet_replica.cc:330] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: stopping tablet replica
I20260806 16:04:03.374528 17374 raft_consensus.cc:2229] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:03.374577 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 16538
I20260806 16:04:03.381261 17376 tablet_replica.cc:330] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: stopping tablet replica
I20260806 16:04:03.381392 17374 raft_consensus.cc:2258] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:03.381474 17376 raft_consensus.cc:2229] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:03.381700 17376 raft_consensus.cc:2258] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:03.382681 17376 ts_tablet_manager.cc:1905] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:03.384159 17374 ts_tablet_manager.cc:1905] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:03.386705 17374 ts_tablet_manager.cc:1918] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:04:03.386781 17376 ts_tablet_manager.cc:1918] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 16:04:03.386818 17374 log.cc:1196] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/fcae30d832414116a73d903f3877e54f
I20260806 16:04:03.386941 17376 log.cc:1196] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/fcae30d832414116a73d903f3877e54f
I20260806 16:04:03.387195 17374 ts_tablet_manager.cc:1939] T fcae30d832414116a73d903f3877e54f P 61edb70fbb7a4716bb47d6797b29306e: Deleting consensus metadata
I20260806 16:04:03.387310 17376 ts_tablet_manager.cc:1939] T fcae30d832414116a73d903f3877e54f P 48f337453cdf494995511f074951c8fa: Deleting consensus metadata
I20260806 16:04:03.388101 16420 catalog_manager.cc:4841] TS 61edb70fbb7a4716bb47d6797b29306e (127.13.148.3:43467): tablet fcae30d832414116a73d903f3877e54f (table test-table [id=fcba2d23d3ff4e349a20054159ef63c7]) successfully deleted
I20260806 16:04:03.388168 16422 catalog_manager.cc:4841] TS 48f337453cdf494995511f074951c8fa (127.13.148.2:35117): tablet fcae30d832414116a73d903f3877e54f (table test-table [id=fcba2d23d3ff4e349a20054159ef63c7]) successfully deleted
W20260806 16:04:03.393261 16420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:46595 (error 108)
I20260806 16:04:03.393707 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 16728
I20260806 16:04:03.401926 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:03.400718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46595 (local address 127.0.0.1:44966)
0806 16:04:03.401592 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:03.401729 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:46595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":399,"threads_started":1}
W20260806 16:04:03.402149 16420 catalog_manager.cc:4584] TS b5456ab5278b40488b27912b3556de04 (127.13.148.1:46595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcae30d832414116a73d903f3877e54f: Network error: Client connection negotiation failed: client connection to 127.13.148.1:46595: connect: Connection refused (error 111)
I20260806 16:04:03.411957 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 16928
I20260806 16:04:03.426313 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 16397
2026-08-06T16:04:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6960 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:03 dist-test-slave-7kzw krb5kdc[17385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:03 dist-test-slave-7kzw krb5kdc[17385](info): set up 2 sockets
Aug 06 16:04:03 dist-test-slave-7kzw krb5kdc[17385](info): commencing operation
krb5kdc: starting...
W20260806 16:04:04.664112 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:04 dist-test-slave-7kzw krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:04.802420 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:33241
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:33241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:04.925571 17401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:04.925952 17401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:04.926034 17401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:04.930627 17401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:04.930747 17401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:04.930784 17401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:04.930822 17401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:04.930858 17401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:04.936419 17401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:33241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:33241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:04.937641 17401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:04.938838 17401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:04.948400 17407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:04.948411 17412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:04.948511 17401 server_base.cc:1034] running on GCE node
W20260806 16:04:04.948459 17409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:04.949400 17401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:04.950738 17401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:04.951924 17401 hybrid_clock.cc:648] HybridClock initialized: now 1786032244951907 us; error 30 us; skew 500 ppm
Aug 06 16:04:04 dist-test-slave-7kzw krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032244, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:04.955142 17401 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:04.956168 17401 webserver.cc:460] Webserver started at http://127.13.148.62:42057/ using document root <none> and password file <none>
I20260806 16:04:04.956820 17401 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:04.956890 17401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:04.957118 17401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:04.959400 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "9d44323da3af46cf99f365c1ff88a956"
format_stamp: "Formatted at 2026-08-06 16:04:04 on dist-test-slave-7kzw"
server_key: "c7b06f69a4af4a52b4e031a17adbfbb5"
server_key_iv: "5dedfe2e995b0add6bae4c32b36b1f8b"
server_key_version: "encryptionkey@0"
I20260806 16:04:04.960045 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "9d44323da3af46cf99f365c1ff88a956"
format_stamp: "Formatted at 2026-08-06 16:04:04 on dist-test-slave-7kzw"
server_key: "c7b06f69a4af4a52b4e031a17adbfbb5"
server_key_iv: "5dedfe2e995b0add6bae4c32b36b1f8b"
server_key_version: "encryptionkey@0"
I20260806 16:04:04.964056 17401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:04.966607 17425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:04.967913 17401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:04:04.968168 17401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "9d44323da3af46cf99f365c1ff88a956"
format_stamp: "Formatted at 2026-08-06 16:04:04 on dist-test-slave-7kzw"
server_key: "c7b06f69a4af4a52b4e031a17adbfbb5"
server_key_iv: "5dedfe2e995b0add6bae4c32b36b1f8b"
server_key_version: "encryptionkey@0"
I20260806 16:04:04.968284 17401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:04.979638 17401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:04.982612 17401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:04.982813 17401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:04.990561 17401 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:33241
I20260806 16:04:04.990650 17493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:33241 every 8 connection(s)
I20260806 16:04:04.991616 17401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:04.994824 17494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:05.000722 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 17401
I20260806 16:04:05.000857 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:05.001166 13904 external_mini_cluster.cc:1428] Setting key ed9a45438e8560789eca1b8b50f1d19f
I20260806 16:04:05.001154 17494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956: Bootstrap starting.
I20260806 16:04:05.004014 17494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:05.005041 17494 log.cc:824] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:05.007647 17494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956: No bootstrap required, opened a new log
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:05.012535 17494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44323da3af46cf99f365c1ff88a956" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33241 } }
I20260806 16:04:05.012831 17494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:05.012902 17494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d44323da3af46cf99f365c1ff88a956, State: Initialized, Role: FOLLOWER
I20260806 16:04:05.013427 17494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44323da3af46cf99f365c1ff88a956" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33241 } }
I20260806 16:04:05.013556 17494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:05.013661 17494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:05.013799 17494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:05.014928 17494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44323da3af46cf99f365c1ff88a956" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33241 } }
I20260806 16:04:05.015321 17494 leader_election.cc:302] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d44323da3af46cf99f365c1ff88a956; no voters:
I20260806 16:04:05.015699 17494 leader_election.cc:288] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:05.015784 17501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:05.016137 17501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [term 1 LEADER]: Becoming Leader. State: Replica: 9d44323da3af46cf99f365c1ff88a956, State: Running, Role: LEADER
I20260806 16:04:05.016472 17501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44323da3af46cf99f365c1ff88a956" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33241 } }
I20260806 16:04:05.017289 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:05.002688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.13.148.62:33241)
0806 16:04:05.003131 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:05.003148 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:05.003185 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:04:05.003824 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:05.003842 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:05.004215 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:05.004555 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:05.005620 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.006839 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.008110 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.008578 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.008702 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:05.011787 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:05.011793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:05.011809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:05.011838 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:05.013959 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:05.014417 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:05.014424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:05.014429 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:05.014609 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:05.014918 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:05.014921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:05.014923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:05.015394 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:05.015606 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:05.015937 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 16:04:05.016275 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":140,"threads_started":1}
I20260806 16:04:05.017321 17494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:05.019064 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d44323da3af46cf99f365c1ff88a956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44323da3af46cf99f365c1ff88a956" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33241 } } }
I20260806 16:04:05.019389 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:05.019614 17502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d44323da3af46cf99f365c1ff88a956. Latest consensus state: current_term: 1 leader_uuid: "9d44323da3af46cf99f365c1ff88a956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44323da3af46cf99f365c1ff88a956" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33241 } } }
I20260806 16:04:05.020340 17502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956 [sys.catalog]: This master's current role is: LEADER
W20260806 16:04:05.023030 17519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:04:05.023134 17519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:04:05.023211 17520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:05.023952 17520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:05.029815 17520 catalog_manager.cc:1347] Generated new cluster ID: 7e4fc4b120654214883dc35165504b30
I20260806 16:04:05.029943 17520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:05.037885 17520 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:05.039258 17520 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:05.049608 17520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d44323da3af46cf99f365c1ff88a956: Generated new TSK 0
I20260806 16:04:05.050540 17520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:04:05.083666 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33241
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:05.202225 17530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:05.202579 17530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:05.202633 17530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:05.206893 17530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:05.206961 17530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:05.207021 17530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:05.212545 17530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:05.213688 17530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:05.214793 17530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:05.225059 17540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:05.225085 17542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:05.225256 17530 server_base.cc:1034] running on GCE node
W20260806 16:04:05.225334 17539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:05.225915 17530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:05.226706 17530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:05.227909 17530 hybrid_clock.cc:648] HybridClock initialized: now 1786032245227891 us; error 34 us; skew 500 ppm
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:05.230953 17530 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:04:05.231851 17530 webserver.cc:460] Webserver started at http://127.13.148.1:37825/ using document root <none> and password file <none>
I20260806 16:04:05.232400 17530 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:05.232462 17530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:05.232649 17530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:05.234849 17530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "84b0b8a5defa4511b181b6c92fac727c"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "5cc53ddddb3cb127a4b619c8a28df7d2"
server_key_iv: "1ce1dda736672caf523cc9ca131094f2"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.235422 17530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "84b0b8a5defa4511b181b6c92fac727c"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "5cc53ddddb3cb127a4b619c8a28df7d2"
server_key_iv: "1ce1dda736672caf523cc9ca131094f2"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.239398 17530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 16:04:05.242034 17550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.243690 17530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:04:05.243857 17530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "84b0b8a5defa4511b181b6c92fac727c"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "5cc53ddddb3cb127a4b619c8a28df7d2"
server_key_iv: "1ce1dda736672caf523cc9ca131094f2"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.243994 17530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:05.255446 17530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:05.259037 17530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:05.259277 17530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:05.259966 17530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:05.260881 17530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:05.260934 17530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.261008 17530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:05.261044 17530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.270954 17530 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:41551
I20260806 16:04:05.270975 17708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:41551 every 8 connection(s)
I20260806 16:04:05.272135 17530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:05.282388 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 17530
I20260806 16:04:05.282552 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:05.282929 13904 external_mini_cluster.cc:1428] Setting key 76ef17f7f1169b0d8e9c33e288a7ddf8
I20260806 16:04:05.287914 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:05.274266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:52207 (local address 127.13.148.62:33241)
0806 16:04:05.274438 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:05.274444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:05.275486 (+ 1042us) server_negotiation.cc:407] Connection header received
0806 16:04:05.276892 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:05.276895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:05.276944 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:05.277025 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:05.278317 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.278915 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.280150 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.280396 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.280488 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:05.283545 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:05.283548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:05.283550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:05.283576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:05.285135 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:05.285882 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:05.285891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:05.285892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:05.285984 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:05.286420 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:05.286427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:05.286432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:05.286752 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:05.286872 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:05.287558 (+ 686us) server_negotiation.cc:299] Negotiation successful
0806 16:04:05.287712 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 16:04:05.288856 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:05.274780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:33241 (local address 127.13.148.1:52207)
0806 16:04:05.275309 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:05.275357 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:05.276668 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:05.277224 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:05.277231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:05.277631 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:05.278150 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:05.278165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.279042 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.279046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:05.280018 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:05.280028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.280546 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.280549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:05.280711 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:05.281402 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:05.281420 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:05.283348 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:05.285267 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:05.285278 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:05.285298 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:05.285744 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:05.286089 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:05.286095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:05.286100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:05.286309 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:05.286944 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:05.286951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:05.287288 (+ 337us) client_negotiation.cc:769] Sending connection context
0806 16:04:05.287511 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 16:04:05.287945 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":125,"threads_started":1}
I20260806 16:04:05.290344 17710 heartbeater.cc:347] Connected to a master server at 127.13.148.62:33241
I20260806 16:04:05.290684 17710 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:05.291271 17710 heartbeater.cc:511] Master 127.13.148.62:33241 requested a full tablet report, sending...
I20260806 16:04:05.293013 17446 ts_manager.cc:194] Registered new tserver with Master: 84b0b8a5defa4511b181b6c92fac727c (127.13.148.1:41551)
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:04:05.294281 17446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:52207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:04:05.314769 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33241
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:05.432736 17715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:05.433046 17715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:05.433131 17715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:05.437247 17715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:05.437345 17715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:05.437438 17715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:05.442886 17715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:05.444016 17715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:05.445065 17715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:05.455332 17724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:05.455332 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:05.455359 17715 server_base.cc:1034] running on GCE node
W20260806 16:04:05.455374 17726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:05.456615 17715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:05.457502 17715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:05.458688 17715 hybrid_clock.cc:648] HybridClock initialized: now 1786032245458651 us; error 58 us; skew 500 ppm
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:05.461933 17715 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:04:05.462956 17715 webserver.cc:460] Webserver started at http://127.13.148.2:39875/ using document root <none> and password file <none>
I20260806 16:04:05.463554 17715 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:05.463655 17715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:05.463919 17715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:05.466081 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "77bb20d49cef449f9119410f9c5314b2"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "b573650cf5bf9406d691cb279a59ddc5"
server_key_iv: "8cae8bf42e1f88a6bdddd83476a30f57"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.466737 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "77bb20d49cef449f9119410f9c5314b2"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "b573650cf5bf9406d691cb279a59ddc5"
server_key_iv: "8cae8bf42e1f88a6bdddd83476a30f57"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.470938 17715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:04:05.473832 17735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.475270 17715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:05.475435 17715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "77bb20d49cef449f9119410f9c5314b2"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "b573650cf5bf9406d691cb279a59ddc5"
server_key_iv: "8cae8bf42e1f88a6bdddd83476a30f57"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.475561 17715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:05.514961 17715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:05.519006 17715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:05.519233 17715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:05.520025 17715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:05.520924 17715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:05.520999 17715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.521085 17715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:05.521139 17715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.530499 17715 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:35711
I20260806 16:04:05.530588 17897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:35711 every 8 connection(s)
I20260806 16:04:05.531502 17715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:05.533751 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 17715
I20260806 16:04:05.533869 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:05.534188 13904 external_mini_cluster.cc:1428] Setting key 9f594f26df95be2cfcbbe10db073f7ef
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:04:05.547585 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:05.533638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33291 (local address 127.13.148.62:33241)
0806 16:04:05.533819 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:05.533827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:05.535807 (+ 1980us) server_negotiation.cc:407] Connection header received
0806 16:04:05.537338 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:05.537345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:05.537415 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:05.537520 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:05.538865 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.539400 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.540570 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.540809 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.540923 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:05.543534 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:05.543540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:05.543547 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:05.543573 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:05.545099 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:05.545704 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:05.545708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:05.545711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:05.545830 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:05.546194 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:05.546197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:05.546198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:05.546425 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:05.546641 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:05.547310 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 16:04:05.547440 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260806 16:04:05.548599 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:05.534122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:33241 (local address 127.13.148.2:33291)
0806 16:04:05.535632 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:05.535693 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:05.537078 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:05.537727 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:05.537735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:05.538110 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:05.538711 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:05.538728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.539538 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.539546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:05.540426 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:05.540436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.540966 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.540970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:05.541140 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:05.541799 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:05.541817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:05.543350 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:05.545237 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:05.545243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:05.545261 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:05.545543 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:05.545944 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:05.545946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:05.545948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:05.546088 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:05.546702 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:05.546712 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:05.547003 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 16:04:05.547188 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 16:04:05.547482 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1272,"thread_start_us":1101,"threads_started":1}
I20260806 16:04:05.550168 17898 heartbeater.cc:347] Connected to a master server at 127.13.148.62:33241
I20260806 16:04:05.550455 17898 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:05.551134 17898 heartbeater.cc:511] Master 127.13.148.62:33241 requested a full tablet report, sending...
I20260806 16:04:05.552668 17446 ts_manager.cc:194] Registered new tserver with Master: 77bb20d49cef449f9119410f9c5314b2 (127.13.148.2:35711)
I20260806 16:04:05.553555 17446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:33291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:04:05.565989 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33241
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:05.686012 17906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:05.686296 17906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:05.686340 17906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:05.690510 17906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:05.690593 17906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:05.690655 17906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:05.695983 17906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:05.697031 17906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:05.698042 17906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:05.708087 17916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:05.708153 17914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:05.708130 17919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:05.708441 17906 server_base.cc:1034] running on GCE node
I20260806 16:04:05.708880 17906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:05.709566 17906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:05.710768 17906 hybrid_clock.cc:648] HybridClock initialized: now 1786032245710750 us; error 29 us; skew 500 ppm
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:05.713934 17906 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:04:05.715003 17906 webserver.cc:460] Webserver started at http://127.13.148.3:42835/ using document root <none> and password file <none>
I20260806 16:04:05.715643 17906 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:05.715744 17906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:05.715992 17906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:05.718246 17906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "394ace3e6a234ed393fc557e2351d2e8"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "62f6b74d3b8386632289d996be35283e"
server_key_iv: "950cdb298e4c7b1677517d72c26ba7fe"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.718957 17906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "394ace3e6a234ed393fc557e2351d2e8"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "62f6b74d3b8386632289d996be35283e"
server_key_iv: "950cdb298e4c7b1677517d72c26ba7fe"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.722924 17906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:05.725579 17933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.726871 17906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:05.727021 17906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "394ace3e6a234ed393fc557e2351d2e8"
format_stamp: "Formatted at 2026-08-06 16:04:05 on dist-test-slave-7kzw"
server_key: "62f6b74d3b8386632289d996be35283e"
server_key_iv: "950cdb298e4c7b1677517d72c26ba7fe"
server_key_version: "encryptionkey@0"
I20260806 16:04:05.727142 17906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:05.770555 17906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:05.774304 17906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:05.774581 17906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:05.775241 17906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:05.776118 17906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:05.776338 17906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.776468 17906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:05.776526 17906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:05.786689 17906 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:39251
I20260806 16:04:05.786779 18095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:39251 every 8 connection(s)
I20260806 16:04:05.787688 17906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:05.794596 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 17906
I20260806 16:04:05.794768 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:05.795102 13904 external_mini_cluster.cc:1428] Setting key 48dc9d6711a9ac4908a3f3bc941f0214
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:05.802561 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:05.789684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58631 (local address 127.13.148.62:33241)
0806 16:04:05.789857 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:05.789861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:05.790837 (+ 976us) server_negotiation.cc:407] Connection header received
0806 16:04:05.792160 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:05.792163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:05.792227 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:05.792333 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:05.793549 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.794099 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.795476 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.795700 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.795816 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:05.798697 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:05.798701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:05.798704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:05.798731 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:05.800304 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:05.800913 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:05.800919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:05.800926 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:05.801012 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:05.801382 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:05.801385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:05.801387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:05.801584 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:05.801695 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:05.802218 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 16:04:05.802349 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 16:04:05.803465 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:05.790057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:33241 (local address 127.13.148.3:58631)
0806 16:04:05.790676 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:05.790722 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:05.791931 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:05.792518 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:05.792525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:05.792904 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:05.793387 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:05.793400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.794231 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.794234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:05.795334 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:05.795344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:05.795856 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:05.795860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:05.796035 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:05.796733 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:05.796753 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:05.798444 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:05.800455 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:05.800461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:05.800479 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:05.800793 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:05.801110 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:05.801114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:05.801119 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:05.801276 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:05.801694 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:05.801703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:05.801971 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 16:04:05.802164 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 16:04:05.802532 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":126,"threads_started":1}
I20260806 16:04:05.804762 18096 heartbeater.cc:347] Connected to a master server at 127.13.148.62:33241
I20260806 16:04:05.804996 18096 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:05.805454 18096 heartbeater.cc:511] Master 127.13.148.62:33241 requested a full tablet report, sending...
I20260806 16:04:05.806592 17446 ts_manager.cc:194] Registered new tserver with Master: 394ace3e6a234ed393fc557e2351d2e8 (127.13.148.3:39251)
I20260806 16:04:05.807229 17446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:58631
I20260806 16:04:05.809819 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 16:04:05.827970 17498 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:05.826013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.13.148.62:33241)
0806 16:04:05.826242 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:05.826248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:05.826410 (+ 162us) server_negotiation.cc:407] Connection header received
0806 16:04:05.826667 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:05.826670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:05.826720 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:05.826825 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:05.827791 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60850: BlockingRecv error: recv got EOF from 127.0.0.1:60850 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 16:04:05.829411 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 17530
I20260806 16:04:05.849236 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 17715
I20260806 16:04:05.859740 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 17906
I20260806 16:04:05.872941 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 17401
2026-08-06T16:04:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2446 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[18111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[18111](info): set up 2 sockets
Aug 06 16:04:05 dist-test-slave-7kzw krb5kdc[18111](info): commencing operation
krb5kdc: starting...
W20260806 16:04:07.104185 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:07 dist-test-slave-7kzw krb5kdc[18111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:07.247004 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46149
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:07.364493 18128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:07.364840 18128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:07.364918 18128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:07.369163 18128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:07.369270 18128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:07.369309 18128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:07.369349 18128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:07.369385 18128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:07.374907 18128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:07.376107 18128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:07.377226 18128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:07.386251 18137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:07.386261 18134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:07.386313 18135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:07.386401 18128 server_base.cc:1034] running on GCE node
I20260806 16:04:07.387166 18128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:07.388422 18128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:07.389737 18128 hybrid_clock.cc:648] HybridClock initialized: now 1786032247389712 us; error 36 us; skew 500 ppm
Aug 06 16:04:07 dist-test-slave-7kzw krb5kdc[18111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032247, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:07.393016 18128 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:07.394053 18128 webserver.cc:460] Webserver started at http://127.13.148.62:39917/ using document root <none> and password file <none>
I20260806 16:04:07.394743 18128 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:07.394841 18128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:07.395098 18128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:07.397261 18128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "5ae4aa00d28f48ea8b7ecef7937edefd"
format_stamp: "Formatted at 2026-08-06 16:04:07 on dist-test-slave-7kzw"
server_key: "57afb33c70030ff2cacec9361642c91c"
server_key_iv: "bec6d0373dec247bf2cdd895830f1f2f"
server_key_version: "encryptionkey@0"
I20260806 16:04:07.397881 18128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "5ae4aa00d28f48ea8b7ecef7937edefd"
format_stamp: "Formatted at 2026-08-06 16:04:07 on dist-test-slave-7kzw"
server_key: "57afb33c70030ff2cacec9361642c91c"
server_key_iv: "bec6d0373dec247bf2cdd895830f1f2f"
server_key_version: "encryptionkey@0"
I20260806 16:04:07.401839 18128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:04:07.404745 18149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:07.406144 18128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:07.406314 18128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "5ae4aa00d28f48ea8b7ecef7937edefd"
format_stamp: "Formatted at 2026-08-06 16:04:07 on dist-test-slave-7kzw"
server_key: "57afb33c70030ff2cacec9361642c91c"
server_key_iv: "bec6d0373dec247bf2cdd895830f1f2f"
server_key_version: "encryptionkey@0"
I20260806 16:04:07.406448 18128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:07.418435 18128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:07.421773 18128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:07.422004 18128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:07.429577 18128 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46149
I20260806 16:04:07.429899 18225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46149 every 8 connection(s)
I20260806 16:04:07.430660 18128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:07.433617 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:07.435281 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 18128
I20260806 16:04:07.435406 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:07.435695 13904 external_mini_cluster.cc:1428] Setting key 7d8599165a2925d8e0e4e31c3c68e336
I20260806 16:04:07.440182 18227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd: Bootstrap starting.
I20260806 16:04:07.442440 18227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:07.443230 18227 log.cc:824] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:04:07 dist-test-slave-7kzw krb5kdc[18111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:07.445397 18227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd: No bootstrap required, opened a new log
I20260806 16:04:07.449227 18227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae4aa00d28f48ea8b7ecef7937edefd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46149 } }
I20260806 16:04:07.449484 18227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:07.449569 18227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae4aa00d28f48ea8b7ecef7937edefd, State: Initialized, Role: FOLLOWER
I20260806 16:04:07.450106 18227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae4aa00d28f48ea8b7ecef7937edefd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46149 } }
I20260806 16:04:07.450276 18227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:07.450346 18227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:07.450445 18227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:07.451335 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:07.437294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.13.148.62:46149)
0806 16:04:07.437797 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:07.437818 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:07.437856 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:04:07.438626 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:07.438646 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:07.438995 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:07.439364 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:07.440330 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:07.441398 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:07.442374 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:07.443025 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:07.443219 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:07.446028 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:07.446034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:07.446048 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:07.446077 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:07.447983 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:07.448499 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:07.448511 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:07.448515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:07.448691 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:07.449033 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:07.449037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:07.449038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:07.449462 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:07.449708 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:07.450063 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 16:04:07.450428 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":138,"threads_started":1}
I20260806 16:04:07.451457 18227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae4aa00d28f48ea8b7ecef7937edefd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46149 } }
I20260806 16:04:07.451803 18227 leader_election.cc:302] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ae4aa00d28f48ea8b7ecef7937edefd; no voters:
I20260806 16:04:07.452147 18227 leader_election.cc:288] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:07.452242 18235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:07.452411 18235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [term 1 LEADER]: Becoming Leader. State: Replica: 5ae4aa00d28f48ea8b7ecef7937edefd, State: Running, Role: LEADER
I20260806 16:04:07.452693 18235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae4aa00d28f48ea8b7ecef7937edefd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46149 } }
I20260806 16:04:07.453434 18227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:07.453840 18237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ae4aa00d28f48ea8b7ecef7937edefd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae4aa00d28f48ea8b7ecef7937edefd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46149 } } }
I20260806 16:04:07.454591 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:04:07.455237 18237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:07.455097 18239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ae4aa00d28f48ea8b7ecef7937edefd. Latest consensus state: current_term: 1 leader_uuid: "5ae4aa00d28f48ea8b7ecef7937edefd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae4aa00d28f48ea8b7ecef7937edefd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46149 } } }
I20260806 16:04:07.455617 18239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd [sys.catalog]: This master's current role is: LEADER
W20260806 16:04:07.458715 18259 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:04:07.458807 18259 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:04:07.458894 18257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:07.459640 18257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:07.467039 18257 catalog_manager.cc:1347] Generated new cluster ID: ce329eb10fb548b3bd5b2357d4bdb3a2
I20260806 16:04:07.467123 18257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:07.484320 18257 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:07.485531 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:07.479123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.13.148.62:46149)
0806 16:04:07.479260 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:07.479268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:07.479487 (+ 219us) server_negotiation.cc:407] Connection header received
0806 16:04:07.479631 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:07.479634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:07.479692 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:07.479775 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:07.480464 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:07.481292 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:07.482717 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:07.483044 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:07.483160 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:07.484342 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:07.484349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:07.484355 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:07.484967 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:07.484999 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:07.485020 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:07.485083 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:07.485298 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 16:04:07.485371 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 16:04:07.485612 18257 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:07.494236 18257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ae4aa00d28f48ea8b7ecef7937edefd: Generated new TSK 0
I20260806 16:04:07.495077 18257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:04:07.504612 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 18128
2026-08-06T16:04:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1627 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:07 dist-test-slave-7kzw krb5kdc[18278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:07 dist-test-slave-7kzw krb5kdc[18278](info): set up 2 sockets
Aug 06 16:04:07 dist-test-slave-7kzw krb5kdc[18278](info): commencing operation
krb5kdc: starting...
W20260806 16:04:08.764084 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:08 dist-test-slave-7kzw krb5kdc[18278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:08.910631 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:43763
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:09.027990 18291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:09.028445 18291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:09.028528 18291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:09.032895 18291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:09.032990 18291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:09.033025 18291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:09.033068 18291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:09.033103 18291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:09.038751 18291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:43763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:43763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:09.040058 18291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:09.041198 18291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:09.050686 18303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:09.050818 18299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:09.050698 18297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:09.050980 18291 server_base.cc:1034] running on GCE node
I20260806 16:04:09.051692 18291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:09.052965 18291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:09.054190 18291 hybrid_clock.cc:648] HybridClock initialized: now 1786032249054161 us; error 52 us; skew 500 ppm
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:09.062743 18291 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:09.063735 18291 webserver.cc:460] Webserver started at http://127.13.148.62:36559/ using document root <none> and password file <none>
I20260806 16:04:09.064302 18291 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:09.064390 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:09.064574 18291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:09.066769 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "0534a1169b7849acb329fa25fbca678b"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "8276128259596831749ace20d959bbe3"
server_key_iv: "91f9ba57236ed10c4afc85a61df273e7"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.067332 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "0534a1169b7849acb329fa25fbca678b"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "8276128259596831749ace20d959bbe3"
server_key_iv: "91f9ba57236ed10c4afc85a61df273e7"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.071342 18291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:09.073876 18315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.075294 18291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:09.075416 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "0534a1169b7849acb329fa25fbca678b"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "8276128259596831749ace20d959bbe3"
server_key_iv: "91f9ba57236ed10c4afc85a61df273e7"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.075510 18291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:09.095875 18291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:09.099647 18291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:09.099957 18291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:09.108505 18291 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:43763
I20260806 16:04:09.108834 18394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:43763 every 8 connection(s)
I20260806 16:04:09.109686 18291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:09.112743 18396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:09.118443 18396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b: Bootstrap starting.
I20260806 16:04:09.119441 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 18291
I20260806 16:04:09.119612 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:09.119930 13904 external_mini_cluster.cc:1428] Setting key a85c38a87373421b5eb0e40af37391c9
I20260806 16:04:09.121380 18396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:09.122159 18396 log.cc:824] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:09.124481 18396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b: No bootstrap required, opened a new log
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:09.129289 18396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0534a1169b7849acb329fa25fbca678b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43763 } }
I20260806 16:04:09.129567 18396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:09.129652 18396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0534a1169b7849acb329fa25fbca678b, State: Initialized, Role: FOLLOWER
I20260806 16:04:09.130137 18396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0534a1169b7849acb329fa25fbca678b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43763 } }
I20260806 16:04:09.130292 18396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:09.130379 18396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:09.130517 18396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:09.131840 18396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0534a1169b7849acb329fa25fbca678b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43763 } }
I20260806 16:04:09.132210 18396 leader_election.cc:302] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0534a1169b7849acb329fa25fbca678b; no voters:
I20260806 16:04:09.132575 18396 leader_election.cc:288] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:09.132651 18402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:09.132874 18402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [term 1 LEADER]: Becoming Leader. State: Replica: 0534a1169b7849acb329fa25fbca678b, State: Running, Role: LEADER
I20260806 16:04:09.133226 18402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0534a1169b7849acb329fa25fbca678b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43763 } }
I20260806 16:04:09.133988 18396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:09.135324 18404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0534a1169b7849acb329fa25fbca678b. Latest consensus state: current_term: 1 leader_uuid: "0534a1169b7849acb329fa25fbca678b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0534a1169b7849acb329fa25fbca678b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43763 } } }
I20260806 16:04:09.135473 18403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0534a1169b7849acb329fa25fbca678b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0534a1169b7849acb329fa25fbca678b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43763 } } }
I20260806 16:04:09.135553 18403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:09.135612 18404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:09.136013 18412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:09.136123 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:09.121198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.13.148.62:43763)
0806 16:04:09.121660 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:09.121677 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:09.121714 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:04:09.122436 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:09.122456 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:09.122845 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:09.123126 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:09.124053 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.125109 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.126289 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.126847 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.126968 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:09.130247 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:09.130252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:09.130267 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:09.130334 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:09.132270 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.132828 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.132833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.132838 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.133096 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.133575 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.133577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.133578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.133986 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:09.134241 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:09.134564 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 16:04:09.134927 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":139,"threads_started":1}
I20260806 16:04:09.139703 18412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:09.145112 18412 catalog_manager.cc:1347] Generated new cluster ID: 2b1ae17a990847939076e6fcf711704f
I20260806 16:04:09.145196 18412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:09.159291 18412 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:09.160389 18412 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:09.169891 18412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0534a1169b7849acb329fa25fbca678b: Generated new TSK 0
I20260806 16:04:09.170708 18412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:04:09.206624 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43763
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:09.328722 18434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:09.329149 18434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:09.329200 18434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:09.333386 18434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:09.333455 18434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:09.333516 18434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:09.339262 18434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:09.340384 18434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:09.341430 18434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:09.351257 18446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:09.351265 18442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:09.351351 18443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:09.351636 18434 server_base.cc:1034] running on GCE node
I20260806 16:04:09.352218 18434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:09.352911 18434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:09.354120 18434 hybrid_clock.cc:648] HybridClock initialized: now 1786032249354085 us; error 54 us; skew 500 ppm
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:09.357041 18434 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:04:09.357924 18434 webserver.cc:460] Webserver started at http://127.13.148.1:38313/ using document root <none> and password file <none>
I20260806 16:04:09.358515 18434 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:09.358596 18434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:09.358786 18434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:09.360795 18434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "35d81ce869a44bf183363c159196b236"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "095e3ef6cfcc6f86090649a2ed7c1438"
server_key_iv: "e1767b74d4270bf29a23a2a236aa454e"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.361313 18434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "35d81ce869a44bf183363c159196b236"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "095e3ef6cfcc6f86090649a2ed7c1438"
server_key_iv: "e1767b74d4270bf29a23a2a236aa454e"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.365231 18434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 16:04:09.367844 18456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.369199 18434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:04:09.369302 18434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "35d81ce869a44bf183363c159196b236"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "095e3ef6cfcc6f86090649a2ed7c1438"
server_key_iv: "e1767b74d4270bf29a23a2a236aa454e"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.369390 18434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:09.379324 18434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:09.382848 18434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:09.383066 18434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:09.383719 18434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:09.384637 18434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:09.384701 18434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.384768 18434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:09.384800 18434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.394798 18434 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:45735
I20260806 16:04:09.394817 18608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:45735 every 8 connection(s)
I20260806 16:04:09.395767 18434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:09.404990 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 18434
I20260806 16:04:09.405158 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:09.405462 13904 external_mini_cluster.cc:1428] Setting key 237414dce5e645ac232c6388c7563e12
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:09.411510 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:09.397904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:57721 (local address 127.13.148.62:43763)
0806 16:04:09.398082 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:09.398086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:09.399114 (+ 1028us) server_negotiation.cc:407] Connection header received
0806 16:04:09.400464 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:09.400467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:09.400516 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:09.400597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:09.401809 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.402355 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.403589 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.403860 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.403962 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:09.406930 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:09.406935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:09.406940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:09.406973 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:09.408882 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.409679 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.409685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.409692 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.409785 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.410205 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.410209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.410211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.410419 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:09.410601 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:09.411172 (+ 571us) server_negotiation.cc:299] Negotiation successful
0806 16:04:09.411313 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:04:09.412398 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:09.398325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43763 (local address 127.13.148.1:57721)
0806 16:04:09.398946 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:09.399000 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:09.400255 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:09.400795 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:09.400801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:09.401182 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:09.401665 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:09.401679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.402513 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.402520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:09.403440 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:09.403450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.404015 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.404018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:09.404159 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:09.404767 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:09.404783 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:09.406694 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:09.409080 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:09.409085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:09.409100 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:09.409415 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:09.409912 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:09.409915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:09.409918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:09.410084 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:09.410613 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:09.410623 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:09.410899 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 16:04:09.411108 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 16:04:09.411474 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":220,"threads_started":1}
I20260806 16:04:09.413961 18610 heartbeater.cc:347] Connected to a master server at 127.13.148.62:43763
I20260806 16:04:09.414278 18610 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:09.414911 18610 heartbeater.cc:511] Master 127.13.148.62:43763 requested a full tablet report, sending...
I20260806 16:04:09.416662 18339 ts_manager.cc:194] Registered new tserver with Master: 35d81ce869a44bf183363c159196b236 (127.13.148.1:45735)
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:04:09.417856 18339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:57721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:04:09.438709 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43763
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:09.560084 18616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:09.560384 18616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:09.560431 18616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:09.564409 18616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:09.564471 18616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:09.564530 18616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:09.570210 18616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:09.571391 18616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:09.572491 18616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:09.582790 18629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:09.582794 18621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:09.582921 18616 server_base.cc:1034] running on GCE node
W20260806 16:04:09.582790 18623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:09.583639 18616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:09.584295 18616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:09.585510 18616 hybrid_clock.cc:648] HybridClock initialized: now 1786032249585491 us; error 29 us; skew 500 ppm
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:09.588973 18616 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:04:09.589916 18616 webserver.cc:460] Webserver started at http://127.13.148.2:45481/ using document root <none> and password file <none>
I20260806 16:04:09.590569 18616 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:09.590649 18616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:09.590881 18616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:09.593003 18616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "262a996dee9f4071b46fd8e8e96a95b1"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "bd1f3623cd6fbcd85956433c0e298ec9"
server_key_iv: "769a3231b3d82548a7d6d6de66f33b5b"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.593604 18616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "262a996dee9f4071b46fd8e8e96a95b1"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "bd1f3623cd6fbcd85956433c0e298ec9"
server_key_iv: "769a3231b3d82548a7d6d6de66f33b5b"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.597692 18616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 16:04:09.600397 18641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.601775 18616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:09.601949 18616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "262a996dee9f4071b46fd8e8e96a95b1"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "bd1f3623cd6fbcd85956433c0e298ec9"
server_key_iv: "769a3231b3d82548a7d6d6de66f33b5b"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.602079 18616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:09.623636 18616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:09.627348 18616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:09.627571 18616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:09.628211 18616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:09.629091 18616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:09.629173 18616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.629251 18616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:09.629302 18616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.639537 18616 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:40061
I20260806 16:04:09.639554 18797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:40061 every 8 connection(s)
I20260806 16:04:09.640527 18616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:09.647650 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 18616
I20260806 16:04:09.647827 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:09.648155 13904 external_mini_cluster.cc:1428] Setting key 97351c09e74596f2737c69162403a4e3
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:09.657757 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:09.642582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:51101 (local address 127.13.148.62:43763)
0806 16:04:09.642779 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:09.642784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:09.643893 (+ 1109us) server_negotiation.cc:407] Connection header received
0806 16:04:09.645337 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:09.645340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:09.645393 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:09.645499 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:09.646967 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.647567 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.649257 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.649577 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.649753 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:09.652963 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:09.652970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:09.652974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:09.653011 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:09.654928 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.655578 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.655585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.655591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.655702 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.656242 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.656246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.656248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.656534 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:09.656663 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:09.657319 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 16:04:09.657481 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 16:04:09.658631 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:09.642976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43763 (local address 127.13.148.2:51101)
0806 16:04:09.643728 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:09.643778 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:09.645097 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:09.645740 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:09.645748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:09.646145 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:09.646796 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:09.646814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.647748 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.647754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:09.649109 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:09.649121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.649797 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.649822 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:09.650027 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:09.650807 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:09.650835 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:09.652748 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:09.655081 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:09.655086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:09.655103 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:09.655430 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:09.655879 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:09.655883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:09.655886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:09.656113 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:09.656703 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:09.656710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:09.656995 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 16:04:09.657200 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 16:04:09.657508 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":130,"threads_started":1}
I20260806 16:04:09.660022 18798 heartbeater.cc:347] Connected to a master server at 127.13.148.62:43763
I20260806 16:04:09.660341 18798 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:04:09.660938 18798 heartbeater.cc:511] Master 127.13.148.62:43763 requested a full tablet report, sending...
I20260806 16:04:09.662348 18339 ts_manager.cc:194] Registered new tserver with Master: 262a996dee9f4071b46fd8e8e96a95b1 (127.13.148.2:40061)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260806 16:04:09.663295 18339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:51101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:04:09.685506 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43763
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:09.801769 18805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:09.802117 18805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:09.802201 18805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:09.806360 18805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:09.806465 18805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:09.806598 18805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:09.812089 18805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:09.813344 18805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:09.814486 18805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:09.824448 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:09.824453 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:09.824606 18805 server_base.cc:1034] running on GCE node
W20260806 16:04:09.824486 18819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:09.825248 18805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:09.825845 18805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:09.827008 18805 hybrid_clock.cc:648] HybridClock initialized: now 1786032249826985 us; error 45 us; skew 500 ppm
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:09.830025 18805 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:04:09.831041 18805 webserver.cc:460] Webserver started at http://127.13.148.3:33249/ using document root <none> and password file <none>
I20260806 16:04:09.831637 18805 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:09.831715 18805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:09.831943 18805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:09.834106 18805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "ef49d826e33e4f31ab0f25a07368cef3"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "832810a228a1dd41ad2bf59872d0efb0"
server_key_iv: "14f761960468f4295bd072dfe38bc510"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.834743 18805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "ef49d826e33e4f31ab0f25a07368cef3"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "832810a228a1dd41ad2bf59872d0efb0"
server_key_iv: "14f761960468f4295bd072dfe38bc510"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.838738 18805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:04:09.841355 18829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.842944 18805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:09.843072 18805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "ef49d826e33e4f31ab0f25a07368cef3"
format_stamp: "Formatted at 2026-08-06 16:04:09 on dist-test-slave-7kzw"
server_key: "832810a228a1dd41ad2bf59872d0efb0"
server_key_iv: "14f761960468f4295bd072dfe38bc510"
server_key_version: "encryptionkey@0"
I20260806 16:04:09.843232 18805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:09.856673 18805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:09.859901 18805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:09.860116 18805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:09.860738 18805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:09.861608 18805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:09.861706 18805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.861801 18805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:09.861848 18805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:09.871901 18805 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:39811
I20260806 16:04:09.872090 19008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:39811 every 8 connection(s)
I20260806 16:04:09.872939 18805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:09.873881 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 18805
I20260806 16:04:09.873982 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:09.874537 13904 external_mini_cluster.cc:1428] Setting key a9023a88028bf76b8701dfb258fac59a
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:09.888134 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:09.875248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:40619 (local address 127.13.148.62:43763)
0806 16:04:09.875408 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:09.875413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:09.876492 (+ 1079us) server_negotiation.cc:407] Connection header received
0806 16:04:09.877997 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:09.878000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:09.878059 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:09.878134 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:09.879541 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.880108 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.881280 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.881517 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.881635 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:09.884475 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:09.884481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:09.884488 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:09.884515 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:09.885934 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.886492 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.886499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.886503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.886595 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.886957 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.886961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.886963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.887157 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:09.887284 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:09.887839 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 16:04:09.887965 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:04:09.889003 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:09.875757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43763 (local address 127.13.148.3:40619)
0806 16:04:09.876347 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:09.876394 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:09.877762 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:09.878329 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:09.878337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:09.878813 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:09.879390 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:09.879406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.880251 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.880256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:09.881140 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:09.881150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.881678 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.881683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:09.881866 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:09.882537 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:09.882555 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:09.884293 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:09.886058 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:09.886065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:09.886082 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:09.886362 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:09.886701 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:09.886704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:09.886706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:09.886855 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:09.887283 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:09.887290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:09.887573 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 16:04:09.887792 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 16:04:09.888129 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":156,"threads_started":1}
I20260806 16:04:09.890413 19009 heartbeater.cc:347] Connected to a master server at 127.13.148.62:43763
I20260806 16:04:09.890813 19009 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:09.891502 19009 heartbeater.cc:511] Master 127.13.148.62:43763 requested a full tablet report, sending...
I20260806 16:04:09.892727 18339 ts_manager.cc:194] Registered new tserver with Master: ef49d826e33e4f31ab0f25a07368cef3 (127.13.148.3:39811)
I20260806 16:04:09.893417 18339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:40619
I20260806 16:04:09.901368 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 16:04:09 dist-test-slave-7kzw krb5kdc[18278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032249, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:09.932154 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:09.921937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.13.148.62:43763)
0806 16:04:09.922110 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:09.922115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:09.922296 (+ 181us) server_negotiation.cc:407] Connection header received
0806 16:04:09.922445 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:09.922448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:09.922556 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:09.922625 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:09.923472 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.924002 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.924993 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:09.925254 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:09.925378 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:09.928392 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:09.928395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:09.928397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:09.928424 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:09.929947 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.930531 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.930542 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.930544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.930751 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:09.931087 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:09.931091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:09.931093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:09.931374 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:09.931537 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:09.931807 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 16:04:09.931958 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 16:04:09.932617 18339 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46406
I20260806 16:04:09.933542 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 18434
I20260806 16:04:09.953506 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 18616
I20260806 16:04:09.963404 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 18805
I20260806 16:04:09.972421 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 18291
2026-08-06T16:04:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2468 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:10 dist-test-slave-7kzw krb5kdc[19023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:10 dist-test-slave-7kzw krb5kdc[19023](info): set up 2 sockets
Aug 06 16:04:10 dist-test-slave-7kzw krb5kdc[19023](info): commencing operation
krb5kdc: starting...
W20260806 16:04:11.239025 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:11 dist-test-slave-7kzw krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:11.378845 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44647
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:44647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:11.497066 19044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:11.497392 19044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:11.497474 19044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:11.501564 19044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:11.501649 19044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:11.501691 19044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:11.501729 19044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:11.501762 19044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:11.507200 19044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:44647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:11.508385 19044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:11.509444 19044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:11.518450 19054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:11.518419 19057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:11.518499 19044 server_base.cc:1034] running on GCE node
W20260806 16:04:11.518430 19055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:11.519398 19044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:11.520522 19044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:11.521731 19044 hybrid_clock.cc:648] HybridClock initialized: now 1786032251521683 us; error 51 us; skew 500 ppm
Aug 06 16:04:11 dist-test-slave-7kzw krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032251, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:11.525111 19044 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:11.526189 19044 webserver.cc:460] Webserver started at http://127.13.148.62:42469/ using document root <none> and password file <none>
I20260806 16:04:11.526852 19044 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:11.526922 19044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:11.527189 19044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:11.529358 19044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "78bd360e11864a97939882b7f3932441"
format_stamp: "Formatted at 2026-08-06 16:04:11 on dist-test-slave-7kzw"
server_key: "9abbf872759b63ff2a7f590f4d774ead"
server_key_iv: "b5b96f094c0ccc2a0b1e4a686a500baa"
server_key_version: "encryptionkey@0"
I20260806 16:04:11.529964 19044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "78bd360e11864a97939882b7f3932441"
format_stamp: "Formatted at 2026-08-06 16:04:11 on dist-test-slave-7kzw"
server_key: "9abbf872759b63ff2a7f590f4d774ead"
server_key_iv: "b5b96f094c0ccc2a0b1e4a686a500baa"
server_key_version: "encryptionkey@0"
I20260806 16:04:11.533862 19044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:04:11.536777 19067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:11.538128 19044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:11.538326 19044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "78bd360e11864a97939882b7f3932441"
format_stamp: "Formatted at 2026-08-06 16:04:11 on dist-test-slave-7kzw"
server_key: "9abbf872759b63ff2a7f590f4d774ead"
server_key_iv: "b5b96f094c0ccc2a0b1e4a686a500baa"
server_key_version: "encryptionkey@0"
I20260806 16:04:11.538465 19044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:11.555327 19044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:11.558691 19044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:11.558919 19044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:11.566550 19044 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:44647
I20260806 16:04:11.566680 19137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:44647 every 8 connection(s)
I20260806 16:04:11.567634 19044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:11.570806 19139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:11.576947 19139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441: Bootstrap starting.
I20260806 16:04:11.577077 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 19044
I20260806 16:04:11.577207 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:11.577527 13904 external_mini_cluster.cc:1428] Setting key b091d2585fb149d500557325675d6487
I20260806 16:04:11.579967 19139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:11.580821 19139 log.cc:824] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:11.583063 19139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441: No bootstrap required, opened a new log
Aug 06 16:04:11 dist-test-slave-7kzw krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:11.588145 19139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd360e11864a97939882b7f3932441" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44647 } }
I20260806 16:04:11.588431 19139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:11.588506 19139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bd360e11864a97939882b7f3932441, State: Initialized, Role: FOLLOWER
I20260806 16:04:11.589113 19139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd360e11864a97939882b7f3932441" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44647 } }
I20260806 16:04:11.589319 19139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:11.589432 19139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:11.589551 19139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:11.590632 19139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd360e11864a97939882b7f3932441" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44647 } }
I20260806 16:04:11.591003 19139 leader_election.cc:302] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78bd360e11864a97939882b7f3932441; no voters:
I20260806 16:04:11.591377 19139 leader_election.cc:288] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:11.591495 19147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:11.591717 19147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [term 1 LEADER]: Becoming Leader. State: Replica: 78bd360e11864a97939882b7f3932441, State: Running, Role: LEADER
I20260806 16:04:11.592039 19147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd360e11864a97939882b7f3932441" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44647 } }
I20260806 16:04:11.592871 19139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:11.593343 19147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78bd360e11864a97939882b7f3932441. Latest consensus state: current_term: 1 leader_uuid: "78bd360e11864a97939882b7f3932441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd360e11864a97939882b7f3932441" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44647 } } }
I20260806 16:04:11.593475 19147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:11.593817 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78bd360e11864a97939882b7f3932441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd360e11864a97939882b7f3932441" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44647 } } }
I20260806 16:04:11.593933 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:11.594064 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:11.578996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.13.148.62:44647)
0806 16:04:11.579412 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:11.579445 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:11.579483 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:04:11.580218 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:11.580238 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:11.580577 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:11.580863 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:11.581869 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:11.582921 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:11.584162 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:11.584708 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:11.584844 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:11.588045 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:11.588051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:11.588066 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:11.588097 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:11.590134 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:11.590652 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:11.590658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:11.590664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:11.590835 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:11.591149 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:11.591152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:11.591153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:11.591726 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:11.592530 (+ 804us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:11.592685 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 16:04:11.592997 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":139,"threads_started":1}
I20260806 16:04:11.594641 19152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:11.598145 19152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:11.604322 19152 catalog_manager.cc:1347] Generated new cluster ID: aa266b6b50ed4158a7c5f0f959da9307
I20260806 16:04:11.604408 19152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:11.615079 19152 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:11.616030 19152 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:11.626087 19152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78bd360e11864a97939882b7f3932441: Generated new TSK 0
I20260806 16:04:11.626904 19152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:04:11.667452 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44647
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:11.785944 19173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:11.786242 19173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:11.786286 19173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:11.790364 19173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:11.790437 19173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:11.790514 19173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:11.796178 19173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:11.797267 19173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:11.798282 19173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:11.808368 19180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:11.808347 19184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:11.808347 19179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:11.808746 19173 server_base.cc:1034] running on GCE node
I20260806 16:04:11.809245 19173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:11.809960 19173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:11.811152 19173 hybrid_clock.cc:648] HybridClock initialized: now 1786032251811124 us; error 45 us; skew 500 ppm
Aug 06 16:04:11 dist-test-slave-7kzw krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032251, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:11.818370 19173 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:04:11.819491 19173 webserver.cc:460] Webserver started at http://127.13.148.1:42231/ using document root <none> and password file <none>
I20260806 16:04:11.820127 19173 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:11.820243 19173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:11.820501 19173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:11.822841 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "24786b9a51a7451cb315818ddb69c083"
format_stamp: "Formatted at 2026-08-06 16:04:11 on dist-test-slave-7kzw"
server_key: "e71ce7e3968705dc10157092be28f745"
server_key_iv: "d205cb15a1cfbe71a2daaaad741be1d3"
server_key_version: "encryptionkey@0"
I20260806 16:04:11.823485 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "24786b9a51a7451cb315818ddb69c083"
format_stamp: "Formatted at 2026-08-06 16:04:11 on dist-test-slave-7kzw"
server_key: "e71ce7e3968705dc10157092be28f745"
server_key_iv: "d205cb15a1cfbe71a2daaaad741be1d3"
server_key_version: "encryptionkey@0"
I20260806 16:04:11.827613 19173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:04:11.830229 19193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:11.831619 19173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:04:11.831784 19173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "24786b9a51a7451cb315818ddb69c083"
format_stamp: "Formatted at 2026-08-06 16:04:11 on dist-test-slave-7kzw"
server_key: "e71ce7e3968705dc10157092be28f745"
server_key_iv: "d205cb15a1cfbe71a2daaaad741be1d3"
server_key_version: "encryptionkey@0"
I20260806 16:04:11.831924 19173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:11.856161 19173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:11.859509 19173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:11.859741 19173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:11.860399 19173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:11.861303 19173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:11.861378 19173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:11.861454 19173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:11.861506 19173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:11.871819 19173 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:41905
I20260806 16:04:11.871871 19361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:41905 every 8 connection(s)
I20260806 16:04:11.872853 19173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:11.876927 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 19173
I20260806 16:04:11.877117 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:11.877496 13904 external_mini_cluster.cc:1428] Setting key cd36cdc9bcad2ff63a3f5ab89402dd6f
Aug 06 16:04:11 dist-test-slave-7kzw krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032251, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:04:11.890364 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:11.875048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:46603 (local address 127.13.148.62:44647)
0806 16:04:11.875244 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:11.875248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:11.876153 (+ 905us) server_negotiation.cc:407] Connection header received
0806 16:04:11.878229 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:11.878233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:11.878323 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:11.878401 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:11.880310 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:11.880900 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:11.882617 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:11.882855 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:11.882997 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:11.886204 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:11.886215 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:11.886218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:11.886247 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:11.887976 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:11.888598 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:11.888606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:11.888608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:11.888691 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:11.889106 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:11.889109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:11.889110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:11.889337 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:11.889434 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:11.890054 (+ 620us) server_negotiation.cc:299] Negotiation successful
0806 16:04:11.890221 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:04:11.891340 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:11.875433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44647 (local address 127.13.148.1:46603)
0806 16:04:11.875977 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:11.876026 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:11.877951 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:11.878664 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:11.878673 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:11.879327 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:11.880058 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:11.880085 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:11.881066 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:11.881073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:11.882387 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:11.882398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:11.883042 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:11.883046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:11.883296 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:11.884038 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:11.884055 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:11.885976 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:11.888125 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:11.888132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:11.888151 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:11.888461 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:11.888838 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:11.888840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:11.888842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:11.888992 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:11.889453 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:11.889459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:11.889780 (+ 321us) client_negotiation.cc:769] Sending connection context
0806 16:04:11.889960 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 16:04:11.890331 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":125,"threads_started":1}
I20260806 16:04:11.892894 19362 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44647
I20260806 16:04:11.893187 19362 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:11.893875 19362 heartbeater.cc:511] Master 127.13.148.62:44647 requested a full tablet report, sending...
I20260806 16:04:11.895779 19091 ts_manager.cc:194] Registered new tserver with Master: 24786b9a51a7451cb315818ddb69c083 (127.13.148.1:41905)
I20260806 16:04:11.897730 19091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:46603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:04:11.914170 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44647
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:12.050655 19369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:12.050948 19369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:12.050995 19369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:12.055284 19369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:12.055351 19369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:12.055408 19369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:12.060458 19369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:12.061583 19369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:12.062575 19369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:12.072253 19378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:12.072325 19369 server_base.cc:1034] running on GCE node
W20260806 16:04:12.072310 19380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:12.072265 19377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:12.072903 19369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:12.073525 19369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:12.074723 19369 hybrid_clock.cc:648] HybridClock initialized: now 1786032252074688 us; error 51 us; skew 500 ppm
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032252, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:12.077580 19369 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:04:12.078383 19369 webserver.cc:460] Webserver started at http://127.13.148.2:40715/ using document root <none> and password file <none>
I20260806 16:04:12.078960 19369 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:12.079025 19369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:12.079198 19369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:12.081158 19369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "8e1863e9ff8d4309a4756c108dcffafe"
format_stamp: "Formatted at 2026-08-06 16:04:12 on dist-test-slave-7kzw"
server_key: "6926bcc7037f38e2636166c33bb6fe8f"
server_key_iv: "354801eb0b5d1f80b85043d837117584"
server_key_version: "encryptionkey@0"
I20260806 16:04:12.081683 19369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "8e1863e9ff8d4309a4756c108dcffafe"
format_stamp: "Formatted at 2026-08-06 16:04:12 on dist-test-slave-7kzw"
server_key: "6926bcc7037f38e2636166c33bb6fe8f"
server_key_iv: "354801eb0b5d1f80b85043d837117584"
server_key_version: "encryptionkey@0"
I20260806 16:04:12.085378 19369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:04:12.087845 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:12.088994 19369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:04:12.089134 19369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "8e1863e9ff8d4309a4756c108dcffafe"
format_stamp: "Formatted at 2026-08-06 16:04:12 on dist-test-slave-7kzw"
server_key: "6926bcc7037f38e2636166c33bb6fe8f"
server_key_iv: "354801eb0b5d1f80b85043d837117584"
server_key_version: "encryptionkey@0"
I20260806 16:04:12.089264 19369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:12.106607 19369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:12.110209 19369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:12.110430 19369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:12.111136 19369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:12.112035 19369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:12.112109 19369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:12.112190 19369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:12.112233 19369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:12.121376 19369 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:34169
I20260806 16:04:12.121415 19544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:34169 every 8 connection(s)
I20260806 16:04:12.122339 19369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:12.122763 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 19369
I20260806 16:04:12.122897 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:12.123216 13904 external_mini_cluster.cc:1428] Setting key 430c96ed295512c8494b4ce9119cd4a5
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032252, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260806 16:04:12.139770 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:12.124454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:36677 (local address 127.13.148.62:44647)
0806 16:04:12.124603 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:12.124608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:12.125693 (+ 1085us) server_negotiation.cc:407] Connection header received
0806 16:04:12.127606 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:12.127610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:12.127667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:12.127779 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:12.129535 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.130148 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.132100 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.132338 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.132434 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:12.135656 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:12.135662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:12.135665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:12.135697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:12.137346 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:12.137932 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:12.137935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:12.137937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:12.138018 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:12.138405 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:12.138408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:12.138410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:12.138671 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:12.138796 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:12.139459 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 16:04:12.139588 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:04:12.140980 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:12.124949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44647 (local address 127.13.148.2:36677)
0806 16:04:12.125516 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:12.125573 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:12.127304 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:12.127996 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:12.128009 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:12.128588 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:12.129333 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:12.129356 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.130310 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.130317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:12.131945 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:12.131961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.132528 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.132533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:12.132783 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:12.133475 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:12.133501 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:12.135463 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:12.137480 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:12.137487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:12.137506 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:12.137804 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:12.138118 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:12.138121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:12.138124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:12.138293 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:12.138818 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:12.138825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:12.139112 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 16:04:12.139355 (+ 243us) client_negotiation.cc:240] Negotiation successful
0806 16:04:12.139777 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":149,"threads_started":1}
I20260806 16:04:12.142402 19545 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44647
I20260806 16:04:12.142791 19545 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:12.143486 19545 heartbeater.cc:511] Master 127.13.148.62:44647 requested a full tablet report, sending...
I20260806 16:04:12.145013 19091 ts_manager.cc:194] Registered new tserver with Master: 8e1863e9ff8d4309a4756c108dcffafe (127.13.148.2:34169)
I20260806 16:04:12.146010 19091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:36677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:04:12.159950 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44647
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:12.284093 19551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:12.284384 19551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:12.284428 19551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:12.288635 19551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:12.288702 19551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:12.288762 19551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:12.294597 19551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:12.295801 19551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:12.296865 19551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:12.306721 19558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:12.306707 19562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:12.306691 19559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:12.306919 19551 server_base.cc:1034] running on GCE node
I20260806 16:04:12.307482 19551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:12.308239 19551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:12.309459 19551 hybrid_clock.cc:648] HybridClock initialized: now 1786032252309433 us; error 51 us; skew 500 ppm
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032252, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:12.312577 19551 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:04:12.313483 19551 webserver.cc:460] Webserver started at http://127.13.148.3:34961/ using document root <none> and password file <none>
I20260806 16:04:12.314071 19551 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:12.314173 19551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:12.314420 19551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:12.316555 19551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "413e5c69024b435cb46448ab087ce2b1"
format_stamp: "Formatted at 2026-08-06 16:04:12 on dist-test-slave-7kzw"
server_key: "c6541f18783e68fa82f036cc8039a39e"
server_key_iv: "19bdbb434c52221ff5f86b0955a80c9f"
server_key_version: "encryptionkey@0"
I20260806 16:04:12.317130 19551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "413e5c69024b435cb46448ab087ce2b1"
format_stamp: "Formatted at 2026-08-06 16:04:12 on dist-test-slave-7kzw"
server_key: "c6541f18783e68fa82f036cc8039a39e"
server_key_iv: "19bdbb434c52221ff5f86b0955a80c9f"
server_key_version: "encryptionkey@0"
I20260806 16:04:12.321324 19551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:04:12.324012 19572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:12.325332 19551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:04:12.325484 19551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "413e5c69024b435cb46448ab087ce2b1"
format_stamp: "Formatted at 2026-08-06 16:04:12 on dist-test-slave-7kzw"
server_key: "c6541f18783e68fa82f036cc8039a39e"
server_key_iv: "19bdbb434c52221ff5f86b0955a80c9f"
server_key_version: "encryptionkey@0"
I20260806 16:04:12.325608 19551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:12.343336 19551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:12.347086 19551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:12.347359 19551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:12.348153 19551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:12.349143 19551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:12.349234 19551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:12.349319 19551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:12.349370 19551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:12.359337 19551 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:40101
I20260806 16:04:12.359361 19743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:40101 every 8 connection(s)
I20260806 16:04:12.360334 19551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:12.368440 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 19551
I20260806 16:04:12.368619 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:12.368937 13904 external_mini_cluster.cc:1428] Setting key ec7e3532521442d0a8da1ce6aa1389b4
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032252, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:12.375483 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:12.362335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:59177 (local address 127.13.148.62:44647)
0806 16:04:12.362566 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:12.362570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:12.363407 (+ 837us) server_negotiation.cc:407] Connection header received
0806 16:04:12.364866 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:12.364870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:12.364932 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:12.365017 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:12.366264 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.366908 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.368179 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.368515 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.368697 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:12.371782 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:12.371785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:12.371787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:12.371815 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:12.373211 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:12.373852 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:12.373855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:12.373856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:12.374005 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:12.374352 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:12.374354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:12.374356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:12.374626 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:12.374722 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:12.375248 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 16:04:12.375357 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 16:04:12.376345 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:12.362740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44647 (local address 127.13.148.3:59177)
0806 16:04:12.363233 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:12.363277 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:12.364632 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:12.365221 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:12.365228 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:12.365605 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:12.366091 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:12.366104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.367048 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.367057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:12.368031 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:12.368041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.368808 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.368816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:12.369092 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:12.369758 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:12.369781 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:12.371602 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:12.373337 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:12.373342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:12.373356 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:12.373742 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:12.374109 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:12.374111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:12.374112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:12.374260 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:12.374744 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:12.374749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:12.374997 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 16:04:12.375163 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 16:04:12.375438 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":123,"threads_started":1}
I20260806 16:04:12.377589 19744 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44647
I20260806 16:04:12.377882 19744 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:12.378381 19744 heartbeater.cc:511] Master 127.13.148.62:44647 requested a full tablet report, sending...
I20260806 16:04:12.379551 19091 ts_manager.cc:194] Registered new tserver with Master: 413e5c69024b435cb46448ab087ce2b1 (127.13.148.3:40101)
I20260806 16:04:12.380319 19091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:59177
I20260806 16:04:12.383565 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:04:12.413020 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:12.402408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.13.148.1:41905)
0806 16:04:12.402757 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:12.402764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:12.402790 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:04:12.402841 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:12.402847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:12.403046 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:12.403261 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:12.404009 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.404888 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.405819 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.406213 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.406305 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:12.409153 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:12.409164 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:12.409178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:12.409225 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:12.411268 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:12.411692 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:12.411697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:12.411699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:12.411789 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:12.412075 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:12.412080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:12.412081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:12.412365 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:12.412545 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:12.412723 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 16:04:12.412864 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:12.424788 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:12.416306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.13.148.62:44647)
0806 16:04:12.416459 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:12.416463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:12.416536 (+ 73us) server_negotiation.cc:407] Connection header received
0806 16:04:12.416635 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:12.416637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:12.416710 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:12.416785 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:12.417385 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.417968 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.418858 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:12.419089 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:12.419180 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:12.421809 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:12.421812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:12.421814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:12.421840 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:12.423239 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:12.423631 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:12.423633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:12.423634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:12.423739 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:12.423985 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:12.423987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:12.423988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:12.424224 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:12.424328 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:12.424536 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 16:04:12.424634 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 16:04:12.425288 19091 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48856
I20260806 16:04:12.425909 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 19173
I20260806 16:04:12.441998 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 19369
I20260806 16:04:12.451337 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 19551
I20260806 16:04:12.464334 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 19044
2026-08-06T16:04:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2495 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19773](info): set up 2 sockets
Aug 06 16:04:12 dist-test-slave-7kzw krb5kdc[19773](info): commencing operation
krb5kdc: starting...
W20260806 16:04:13.687685 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:13 dist-test-slave-7kzw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:13.829608 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42997
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:42997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:13.949883 19789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:13.950286 19789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:13.950374 19789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:13.954941 19789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:13.955054 19789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:13.955089 19789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:13.955132 19789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:13.955168 19789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:13.960911 19789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:42997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:13.962198 19789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:13.963636 19789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:13.973104 19795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:13.973155 19796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:13.973294 19789 server_base.cc:1034] running on GCE node
W20260806 16:04:13.973444 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:13.974112 19789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:13.975389 19789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:13.976692 19789 hybrid_clock.cc:648] HybridClock initialized: now 1786032253976592 us; error 47 us; skew 500 ppm
Aug 06 16:04:13 dist-test-slave-7kzw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032253, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:13.979714 19789 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:13.980471 19789 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:13.980538 19789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:13.980722 19789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:13.982911 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "a21b36e5276f4c1583810efdb027b558"
format_stamp: "Formatted at 2026-08-06 16:04:13 on dist-test-slave-7kzw"
server_key: "b5164b35e7814a83654753bb80becbde"
server_key_iv: "6b902429e5555e2ac9928421493ed8c8"
server_key_version: "encryptionkey@0"
I20260806 16:04:13.983456 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "a21b36e5276f4c1583810efdb027b558"
format_stamp: "Formatted at 2026-08-06 16:04:13 on dist-test-slave-7kzw"
server_key: "b5164b35e7814a83654753bb80becbde"
server_key_iv: "6b902429e5555e2ac9928421493ed8c8"
server_key_version: "encryptionkey@0"
I20260806 16:04:13.987442 19789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:13.990049 19811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:13.991782 19789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:13.991953 19789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "a21b36e5276f4c1583810efdb027b558"
format_stamp: "Formatted at 2026-08-06 16:04:13 on dist-test-slave-7kzw"
server_key: "b5164b35e7814a83654753bb80becbde"
server_key_iv: "6b902429e5555e2ac9928421493ed8c8"
server_key_version: "encryptionkey@0"
I20260806 16:04:13.992081 19789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:14.005336 19789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:14.009466 19789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:14.009770 19789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:14.017647 19789 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:42997
I20260806 16:04:14.017970 19889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:42997 every 8 connection(s)
I20260806 16:04:14.018411 19789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:14.021502 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:14.027695 19890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558: Bootstrap starting.
I20260806 16:04:14.027817 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 19789
I20260806 16:04:14.027944 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:14.028250 13904 external_mini_cluster.cc:1428] Setting key 9f3c611fcdab60a94f6d7991aa94e1f4
I20260806 16:04:14.030740 19890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:14.031545 19890 log.cc:824] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:14.033573 19890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558: No bootstrap required, opened a new log
I20260806 16:04:14.037216 19890 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21b36e5276f4c1583810efdb027b558" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42997 } }
I20260806 16:04:14.037501 19890 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 06 16:04:14 dist-test-slave-7kzw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:14.037590 19890 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21b36e5276f4c1583810efdb027b558, State: Initialized, Role: FOLLOWER
I20260806 16:04:14.038149 19890 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21b36e5276f4c1583810efdb027b558" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42997 } }
I20260806 16:04:14.038326 19890 raft_consensus.cc:388] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:14.038410 19890 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:14.038564 19890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:14.040009 19890 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21b36e5276f4c1583810efdb027b558" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42997 } }
I20260806 16:04:14.040426 19890 leader_election.cc:302] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a21b36e5276f4c1583810efdb027b558; no voters:
I20260806 16:04:14.040830 19890 leader_election.cc:288] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:14.040966 19898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:14.041229 19898 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [term 1 LEADER]: Becoming Leader. State: Replica: a21b36e5276f4c1583810efdb027b558, State: Running, Role: LEADER
I20260806 16:04:14.041659 19898 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21b36e5276f4c1583810efdb027b558" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42997 } }
I20260806 16:04:14.042366 19890 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:14.043515 19900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21b36e5276f4c1583810efdb027b558. Latest consensus state: current_term: 1 leader_uuid: "a21b36e5276f4c1583810efdb027b558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21b36e5276f4c1583810efdb027b558" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42997 } } }
I20260806 16:04:14.043658 19900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:14.043999 19899 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a21b36e5276f4c1583810efdb027b558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21b36e5276f4c1583810efdb027b558" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42997 } } }
I20260806 16:04:14.044076 19907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:14.044095 19899 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:14.044757 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.029770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.13.148.62:42997)
0806 16:04:14.030258 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.030275 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.030314 (+ 39us) server_negotiation.cc:407] Connection header received
0806 16:04:14.031204 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.031226 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.031586 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.031976 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:14.032832 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.033900 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.034838 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.035339 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.035523 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.038237 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:14.038244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:14.038264 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:14.038305 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:14.040672 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.041309 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.041318 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.041326 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.041485 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.041864 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.041868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.041869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.042452 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:14.042693 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.043190 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.043569 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":156,"threads_started":1}
I20260806 16:04:14.047518 19907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:14.054250 19907 catalog_manager.cc:1347] Generated new cluster ID: 1abddb296e1f4b12b31a962323ff33d6
I20260806 16:04:14.054339 19907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:14.072578 19907 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:14.073900 19907 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:14.085332 19907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558: Generated new TSK 0
I20260806 16:04:14.086328 19907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:04:14.121968 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42997
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:14.238194 19921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:14.238538 19921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:14.238590 19921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:14.242944 19921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:14.243058 19921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:14.243165 19921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:14.248929 19921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:14.250012 19921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:14.251051 19921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:14.261001 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:14.261003 19936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:14.261003 19927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:14.261117 19921 server_base.cc:1034] running on GCE node
I20260806 16:04:14.262441 19921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:14.263131 19921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:14.264330 19921 hybrid_clock.cc:648] HybridClock initialized: now 1786032254264317 us; error 36 us; skew 500 ppm
Aug 06 16:04:14 dist-test-slave-7kzw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:14.271831 19921 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:04:14.272670 19921 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:14.272773 19921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:14.273021 19921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:14.275240 19921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "6e58b5c6226d4e128a456d7a79e56648"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "b0d88676ad9ce6f39579d2cdde2fbdd7"
server_key_iv: "ccdb8cf088e91a0889d8c169fd5489b6"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.275823 19921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "6e58b5c6226d4e128a456d7a79e56648"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "b0d88676ad9ce6f39579d2cdde2fbdd7"
server_key_iv: "ccdb8cf088e91a0889d8c169fd5489b6"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.279698 19921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:14.282387 19943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.283674 19921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:04:14.283823 19921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "6e58b5c6226d4e128a456d7a79e56648"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "b0d88676ad9ce6f39579d2cdde2fbdd7"
server_key_iv: "ccdb8cf088e91a0889d8c169fd5489b6"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.283941 19921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:14.311717 19921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:14.315296 19921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:14.315515 19921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:14.316124 19921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:14.316993 19921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:14.317078 19921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.317209 19921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:14.317255 19921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.326659 19921 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:42131
I20260806 16:04:14.326684 20115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:42131 every 8 connection(s)
I20260806 16:04:14.327312 19921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:14.329697 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 19921
I20260806 16:04:14.329847 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:14.330161 13904 external_mini_cluster.cc:1428] Setting key 9af2ac5c87b6ccd9bf53f8e7f40597fd
Aug 06 16:04:14 dist-test-slave-7kzw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:04:14.344031 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.329066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:40135 (local address 127.13.148.62:42997)
0806 16:04:14.329209 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.329213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.330147 (+ 934us) server_negotiation.cc:407] Connection header received
0806 16:04:14.331779 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.331783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.331844 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.331952 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:14.333141 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.334047 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.335901 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.336135 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.336230 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.339428 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:14.339434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:14.339439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:14.339478 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:14.341308 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.341893 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.341897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.341899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.342001 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.342521 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.342525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.342526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.342771 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:14.342892 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.343761 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.343893 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:04:14.344959 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.329379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42997 (local address 127.13.148.1:40135)
0806 16:04:14.329994 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:14.330039 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:14.331443 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:14.332104 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:14.332111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:14.332478 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:14.332978 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.332991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.334210 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.334216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.335755 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.335770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.336314 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.336320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.336555 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.337341 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:14.337368 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:14.339234 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:14.341445 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:14.341450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:14.341463 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:14.341751 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:14.342162 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:14.342172 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:14.342173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:14.342350 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:14.342904 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:14.342912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:14.343187 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 16:04:14.343526 (+ 339us) client_negotiation.cc:240] Negotiation successful
0806 16:04:14.343887 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:04:14.346418 20116 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42997
I20260806 16:04:14.346724 20116 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:14.347213 20116 heartbeater.cc:511] Master 127.13.148.62:42997 requested a full tablet report, sending...
I20260806 16:04:14.349073 19833 ts_manager.cc:194] Registered new tserver with Master: 6e58b5c6226d4e128a456d7a79e56648 (127.13.148.1:42131)
I20260806 16:04:14.350277 19833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:40135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:04:14.365789 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42997
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:14.485304 20125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:14.485575 20125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:14.485620 20125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:14.489746 20125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:14.489812 20125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:14.489871 20125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:14.496058 20125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:14.497340 20125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:14.498394 20125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:14.508373 20132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:14.508384 20137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:14.508445 20131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:14.508596 20125 server_base.cc:1034] running on GCE node
I20260806 16:04:14.509224 20125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:14.509848 20125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:14.511051 20125 hybrid_clock.cc:648] HybridClock initialized: now 1786032254511031 us; error 31 us; skew 500 ppm
Aug 06 16:04:14 dist-test-slave-7kzw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:14.514130 20125 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:04:14.514987 20125 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:14.515059 20125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:14.515242 20125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:14.517339 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "d6cfe7c0a86346af974c6becb8578b28"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "46d81a61f7ce1ebb2bc4157f81c1b7b7"
server_key_iv: "0aca5441023f6b9da0cf2f96d5564093"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.517876 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "d6cfe7c0a86346af974c6becb8578b28"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "46d81a61f7ce1ebb2bc4157f81c1b7b7"
server_key_iv: "0aca5441023f6b9da0cf2f96d5564093"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.521775 20125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:04:14.524421 20144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.525995 20125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 16:04:14.526121 20125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "d6cfe7c0a86346af974c6becb8578b28"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "46d81a61f7ce1ebb2bc4157f81c1b7b7"
server_key_iv: "0aca5441023f6b9da0cf2f96d5564093"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.526229 20125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:14.556303 20125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:14.559890 20125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:14.560060 20125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:14.560616 20125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:14.561444 20125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:14.561504 20125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.561545 20125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:14.561561 20125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.570757 20125 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:41339
I20260806 16:04:14.570775 20316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:41339 every 8 connection(s)
I20260806 16:04:14.571415 20125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:14.574656 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 20125
I20260806 16:04:14.574821 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:14.575143 13904 external_mini_cluster.cc:1428] Setting key 6cf2304bdde4349101ee3f55abeb9d9d
Aug 06 16:04:14 dist-test-slave-7kzw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:14.586199 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.573202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39529 (local address 127.13.148.62:42997)
0806 16:04:14.573341 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.573345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.574144 (+ 799us) server_negotiation.cc:407] Connection header received
0806 16:04:14.575807 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.575811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.575872 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.575958 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:14.577251 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.577781 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.579031 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.579272 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.579385 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.582015 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:14.582018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:14.582021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:14.582044 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:14.583790 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.584350 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.584354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.584356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.584444 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.584802 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.584807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.584808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.585042 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:14.585186 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.585937 (+ 751us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.586064 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:04:14.587592 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.573519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42997 (local address 127.13.148.2:39529)
0806 16:04:14.573985 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:14.574029 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:14.575572 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:14.576135 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:14.576142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:14.576507 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:14.577095 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.577109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.577917 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.577920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.578899 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.578909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.579437 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.579442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.579610 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.580284 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:14.580301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:14.581837 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:14.583936 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:14.583942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:14.583958 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:14.584237 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:14.584541 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:14.584545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:14.584547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:14.584701 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:14.585186 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:14.585197 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:14.585561 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 16:04:14.585846 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 16:04:14.586255 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260806 16:04:14.589047 20317 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42997
I20260806 16:04:14.589392 20317 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260806 16:04:14.590003 20317 heartbeater.cc:511] Master 127.13.148.62:42997 requested a full tablet report, sending...
I20260806 16:04:14.591327 19833 ts_manager.cc:194] Registered new tserver with Master: d6cfe7c0a86346af974c6becb8578b28 (127.13.148.2:41339)
I20260806 16:04:14.591964 19833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:39529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:04:14.609177 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42997
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:14.722900 20326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:14.723160 20326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:14.723204 20326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:14.727843 20326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:14.727900 20326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:14.727988 20326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:14.733255 20326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:14.734294 20326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:14.735332 20326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:14.745496 20336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:14.745448 20334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:14.745472 20333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:14.745483 20326 server_base.cc:1034] running on GCE node
I20260806 16:04:14.746311 20326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:14.747025 20326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:14.748235 20326 hybrid_clock.cc:648] HybridClock initialized: now 1786032254748217 us; error 32 us; skew 500 ppm
Aug 06 16:04:14 dist-test-slave-7kzw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:14.751252 20326 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:04:14.751974 20326 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:14.752048 20326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:14.752220 20326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:14.754331 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "0602592c9e324b48b927f22510dcc6f4"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "38477d1f884edb0973b9529dd573bb33"
server_key_iv: "d432ec51bf3e5a2523a28f24adeaaf95"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.755071 20326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "0602592c9e324b48b927f22510dcc6f4"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "38477d1f884edb0973b9529dd573bb33"
server_key_iv: "d432ec51bf3e5a2523a28f24adeaaf95"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.759272 20326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:04:14.761751 20345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.763226 20326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:04:14.763331 20326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "0602592c9e324b48b927f22510dcc6f4"
format_stamp: "Formatted at 2026-08-06 16:04:14 on dist-test-slave-7kzw"
server_key: "38477d1f884edb0973b9529dd573bb33"
server_key_iv: "d432ec51bf3e5a2523a28f24adeaaf95"
server_key_version: "encryptionkey@0"
I20260806 16:04:14.763422 20326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:14.776953 20326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:14.780128 20326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:14.780288 20326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:14.780824 20326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:14.781632 20326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:14.781690 20326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.781741 20326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:14.781764 20326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:14.791747 20326 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:41047
I20260806 16:04:14.791764 20503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:41047 every 8 connection(s)
I20260806 16:04:14.792389 20326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:14.796829 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 20326
I20260806 16:04:14.796980 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:14.797317 13904 external_mini_cluster.cc:1428] Setting key 126d5735a264f123599378b7ff599119
Aug 06 16:04:14 dist-test-slave-7kzw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:14.807404 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.794307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:47473 (local address 127.13.148.62:42997)
0806 16:04:14.794454 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.794498 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.795350 (+ 852us) server_negotiation.cc:407] Connection header received
0806 16:04:14.796951 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.796955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.797025 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.797151 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:14.798562 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.799342 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.800671 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.800920 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.801025 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.803923 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:14.803927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:14.803929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:14.803956 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:14.805315 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.805871 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.805874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.805875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.805952 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.806290 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.806293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.806294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.806550 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:14.806631 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.807150 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.807257 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:04:14.808333 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.794731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42997 (local address 127.13.148.3:47473)
0806 16:04:14.795198 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:14.795245 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:14.796649 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:14.797368 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:14.797378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:14.797874 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:14.798355 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.798369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.799485 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.799488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.800504 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.800514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.801098 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.801101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.801257 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.801899 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:14.801917 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:14.803746 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:14.805439 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:14.805444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:14.805458 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:14.805771 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:14.806032 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:14.806034 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:14.806036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:14.806189 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:14.806654 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:14.806660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:14.806918 (+ 258us) client_negotiation.cc:769] Sending connection context
0806 16:04:14.807090 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 16:04:14.807389 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
I20260806 16:04:14.809808 20504 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42997
I20260806 16:04:14.810119 20504 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:14.810745 20504 heartbeater.cc:511] Master 127.13.148.62:42997 requested a full tablet report, sending...
I20260806 16:04:14.812003 19832 ts_manager.cc:194] Registered new tserver with Master: 0602592c9e324b48b927f22510dcc6f4 (127.13.148.3:41047)
I20260806 16:04:14.812690 19832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:47473
I20260806 16:04:14.825001 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:04:14.835472 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.827267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.13.148.62:42997)
0806 16:04:14.827435 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.827438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.827642 (+ 204us) server_negotiation.cc:407] Connection header received
0806 16:04:14.827700 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.827702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.827750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.827831 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:14.828623 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.829206 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.830121 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.830346 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.830437 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.832019 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:14.832022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:14.832024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:14.832048 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:14.833442 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.834034 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.834038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.834040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.834130 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:14.834544 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:14.834550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:14.834556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:14.834808 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:14.834920 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.835230 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.835341 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:04:14.839123 19832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:14.842254 19832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:14.859647 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.852888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41339 (local address 127.0.0.1:42248)
0806 16:04:14.853678 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:14.853742 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:14.853898 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:14.855080 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:14.855083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:14.855104 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:14.855217 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.855225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.856322 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.856326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.857819 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.857830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.859195 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.859199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.859308 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.859332 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:04:14.859405 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 16:04:14.859473 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260806 16:04:14.859659 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.853278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.13.148.1:42131)
0806 16:04:14.853853 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.853859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.854192 (+ 333us) server_negotiation.cc:407] Connection header received
0806 16:04:14.854380 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.854388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.854658 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.854824 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:14.855230 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.856491 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.858011 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.858831 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.858952 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.859115 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.859346 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.859459 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":97,"threads_started":1}
I20260806 16:04:14.860391 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.853058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.13.148.2:41339)
0806 16:04:14.853665 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.853678 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.853913 (+ 235us) server_negotiation.cc:407] Connection header received
0806 16:04:14.854690 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.854697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.854900 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.855099 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:14.855357 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.856160 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.857943 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.858831 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.858952 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.859115 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.860122 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.860230 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":136,"threads_started":1}
I20260806 16:04:14.860038 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.852975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42131 (local address 127.0.0.1:44400)
0806 16:04:14.854092 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:14.854107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:14.854210 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:14.854851 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:14.854857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:14.854887 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:14.855066 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.855075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.856628 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.856633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.857876 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.857887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.858971 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.858975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.859107 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.859160 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 16:04:14.859300 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 16:04:14.859463 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":29,"thread_start_us":105,"threads_started":1}
I20260806 16:04:14.860626 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.854321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.13.148.3:41047)
0806 16:04:14.854723 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.854732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.855065 (+ 333us) server_negotiation.cc:407] Connection header received
0806 16:04:14.855308 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.855317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.855516 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.855648 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:14.856130 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.857285 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.858595 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.859426 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.859547 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.859693 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.860322 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.860437 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":145,"threads_started":1}
I20260806 16:04:14.860203 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.854131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:41047 (local address 127.0.0.1:52520)
0806 16:04:14.854938 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:14.854954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:14.855145 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:14.855858 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:14.855861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:14.855880 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:14.855986 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.855994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.857436 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.857440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.858420 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.858428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.859766 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.859770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.859885 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.859935 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 16:04:14.860014 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 16:04:14.860103 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":143,"threads_started":1}
I20260806 16:04:14.864338 20223 tablet_service.cc:1467] Processing CreateTablet for tablet b0f692474f754f4e84fcb27db4fa6c6c (DEFAULT_TABLE table=test-table [id=2aba36bab85648e39b6596c5557b6667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:14.864338 20006 tablet_service.cc:1467] Processing CreateTablet for tablet b0f692474f754f4e84fcb27db4fa6c6c (DEFAULT_TABLE table=test-table [id=2aba36bab85648e39b6596c5557b6667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:14.864344 20411 tablet_service.cc:1467] Processing CreateTablet for tablet b0f692474f754f4e84fcb27db4fa6c6c (DEFAULT_TABLE table=test-table [id=2aba36bab85648e39b6596c5557b6667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:14.865588 20411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f692474f754f4e84fcb27db4fa6c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:14.865586 20223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f692474f754f4e84fcb27db4fa6c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:14.865586 20006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f692474f754f4e84fcb27db4fa6c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:14.872531 20527 tablet_bootstrap.cc:492] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: Bootstrap starting.
I20260806 16:04:14.873600 20529 tablet_bootstrap.cc:492] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648: Bootstrap starting.
I20260806 16:04:14.874023 20530 tablet_bootstrap.cc:492] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: Bootstrap starting.
I20260806 16:04:14.875432 20527 tablet_bootstrap.cc:654] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:14.876048 20529 tablet_bootstrap.cc:654] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:14.876047 20530 tablet_bootstrap.cc:654] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:14.876859 20527 log.cc:824] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:14.876992 20529 log.cc:824] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:14.877254 20530 log.cc:824] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:14.879029 20527 tablet_bootstrap.cc:492] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: No bootstrap required, opened a new log
I20260806 16:04:14.879266 20527 ts_tablet_manager.cc:1397] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:04:14.879359 20530 tablet_bootstrap.cc:492] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: No bootstrap required, opened a new log
I20260806 16:04:14.879557 20530 ts_tablet_manager.cc:1397] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:04:14.879812 20529 tablet_bootstrap.cc:492] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648: No bootstrap required, opened a new log
I20260806 16:04:14.879998 20529 ts_tablet_manager.cc:1397] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260806 16:04:14.883204 20527 raft_consensus.cc:348] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:14.883618 20527 raft_consensus.cc:374] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:14.883711 20527 raft_consensus.cc:729] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0602592c9e324b48b927f22510dcc6f4, State: Initialized, Role: FOLLOWER
I20260806 16:04:14.883651 20530 raft_consensus.cc:348] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:14.883946 20530 raft_consensus.cc:374] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:14.884056 20530 raft_consensus.cc:729] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cfe7c0a86346af974c6becb8578b28, State: Initialized, Role: FOLLOWER
I20260806 16:04:14.884357 20527 consensus_queue.cc:260] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:14.884514 20530 consensus_queue.cc:260] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:14.884464 20529 raft_consensus.cc:348] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:14.884773 20529 raft_consensus.cc:374] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:14.884902 20529 raft_consensus.cc:729] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e58b5c6226d4e128a456d7a79e56648, State: Initialized, Role: FOLLOWER
I20260806 16:04:14.885164 20504 heartbeater.cc:503] Master 127.13.148.62:42997 was elected leader, sending a full tablet report...
I20260806 16:04:14.885553 20527 ts_tablet_manager.cc:1428] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:04:14.885560 20530 ts_tablet_manager.cc:1428] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:04:14.885968 20317 heartbeater.cc:503] Master 127.13.148.62:42997 was elected leader, sending a full tablet report...
I20260806 16:04:14.886163 20529 consensus_queue.cc:260] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:14.887240 20529 ts_tablet_manager.cc:1428] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260806 16:04:14.887591 20116 heartbeater.cc:503] Master 127.13.148.62:42997 was elected leader, sending a full tablet report...
I20260806 16:04:14.982865 20536 raft_consensus.cc:482] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:14.983119 20536 raft_consensus.cc:504] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:14.984256 20536 leader_election.cc:288] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6cfe7c0a86346af974c6becb8578b28 (127.13.148.2:41339), 6e58b5c6226d4e128a456d7a79e56648 (127.13.148.1:42131)
I20260806 16:04:14.989892 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.984724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:55091 (local address 127.13.148.1:42131)
0806 16:04:14.984888 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.984893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.984974 (+ 81us) server_negotiation.cc:407] Connection header received
0806 16:04:14.985194 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.985198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.985257 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.985360 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:14.985797 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.986639 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.988541 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.989193 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.989325 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.989382 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.989678 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.989754 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:04:14.989892 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.984709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:36373 (local address 127.13.148.2:41339)
0806 16:04:14.984843 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:14.984848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:14.985161 (+ 313us) server_negotiation.cc:407] Connection header received
0806 16:04:14.985224 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:14.985227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:14.985278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:14.985360 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:14.985816 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.986622 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.988541 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.989186 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.989325 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.989382 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:14.989678 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 16:04:14.989754 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:04:14.989897 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.984604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42131 (local address 127.13.148.3:55091)
0806 16:04:14.984871 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:14.984888 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:14.985016 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:14.985400 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:14.985403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:14.985445 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:14.985603 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.985614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.986793 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.986797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.988398 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.988410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.989337 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.989341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.989484 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.989545 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 16:04:14.989620 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 16:04:14.989711 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":228,"spinlock_wait_cycles":28160}
I20260806 16:04:14.989907 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:14.984604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41339 (local address 127.13.148.3:36373)
0806 16:04:14.985042 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:14.985057 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:14.985150 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:14.985400 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:14.985403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:14.985446 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:14.985637 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.985684 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.986793 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.986797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.988400 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:14.988410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:14.989337 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:14.989341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:14.989481 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:14.989534 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 16:04:14.989620 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 16:04:14.989711 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":264,"spinlock_wait_cycles":8576,"thread_start_us":178,"threads_started":1}
I20260806 16:04:14.990773 20036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f692474f754f4e84fcb27db4fa6c6c" candidate_uuid: "0602592c9e324b48b927f22510dcc6f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58b5c6226d4e128a456d7a79e56648" is_pre_election: true
I20260806 16:04:14.990778 20253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f692474f754f4e84fcb27db4fa6c6c" candidate_uuid: "0602592c9e324b48b927f22510dcc6f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cfe7c0a86346af974c6becb8578b28" is_pre_election: true
I20260806 16:04:14.991104 20036 raft_consensus.cc:2454] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0602592c9e324b48b927f22510dcc6f4 in term 0.
I20260806 16:04:14.991112 20253 raft_consensus.cc:2454] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0602592c9e324b48b927f22510dcc6f4 in term 0.
I20260806 16:04:14.991626 20346 leader_election.cc:302] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0602592c9e324b48b927f22510dcc6f4, 6e58b5c6226d4e128a456d7a79e56648, d6cfe7c0a86346af974c6becb8578b28; no voters:
I20260806 16:04:14.991935 20536 raft_consensus.cc:2781] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:14.992024 20536 raft_consensus.cc:482] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:14.992069 20536 raft_consensus.cc:3037] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:14.993227 20536 raft_consensus.cc:504] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:14.993640 20536 leader_election.cc:288] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [CANDIDATE]: Term 1 election: Requested vote from peers d6cfe7c0a86346af974c6becb8578b28 (127.13.148.2:41339), 6e58b5c6226d4e128a456d7a79e56648 (127.13.148.1:42131)
I20260806 16:04:14.994088 20036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f692474f754f4e84fcb27db4fa6c6c" candidate_uuid: "0602592c9e324b48b927f22510dcc6f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e58b5c6226d4e128a456d7a79e56648"
I20260806 16:04:14.994145 20253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f692474f754f4e84fcb27db4fa6c6c" candidate_uuid: "0602592c9e324b48b927f22510dcc6f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cfe7c0a86346af974c6becb8578b28"
I20260806 16:04:14.994278 20253 raft_consensus.cc:3037] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:14.994256 20036 raft_consensus.cc:3037] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:14.995481 20253 raft_consensus.cc:2454] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0602592c9e324b48b927f22510dcc6f4 in term 1.
I20260806 16:04:14.995647 20036 raft_consensus.cc:2454] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0602592c9e324b48b927f22510dcc6f4 in term 1.
I20260806 16:04:14.995975 20347 leader_election.cc:302] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0602592c9e324b48b927f22510dcc6f4, d6cfe7c0a86346af974c6becb8578b28; no voters:
I20260806 16:04:14.996268 20536 raft_consensus.cc:2781] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:14.996505 20536 raft_consensus.cc:686] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0602592c9e324b48b927f22510dcc6f4, State: Running, Role: LEADER
I20260806 16:04:14.996887 20536 consensus_queue.cc:237] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } }
I20260806 16:04:15.000777 19832 catalog_manager.cc:5495] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0602592c9e324b48b927f22510dcc6f4 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "0602592c9e324b48b927f22510dcc6f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0602592c9e324b48b927f22510dcc6f4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41047 } health_report { overall_health: HEALTHY } } }
I20260806 16:04:15.014612 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.010858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.13.148.3:41047)
0806 16:04:15.010991 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.010995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.011009 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:15.011136 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.011138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.011195 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.011286 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:15.011529 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:15.012024 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:15.013043 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:15.013275 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:15.013398 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:15.013729 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:15.014161 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:15.014298 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.014362 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.014424 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:04:15.020604 20253 raft_consensus.cc:1261] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 1 FOLLOWER]: Refusing update from remote peer 0602592c9e324b48b927f22510dcc6f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:15.020627 20036 raft_consensus.cc:1261] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [term 1 FOLLOWER]: Refusing update from remote peer 0602592c9e324b48b927f22510dcc6f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:15.021340 20536 consensus_queue.cc:1035] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e58b5c6226d4e128a456d7a79e56648" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:15.021571 20541 consensus_queue.cc:1035] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6cfe7c0a86346af974c6becb8578b28" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:15.039240 20546 mvcc.cc:205] Tried to move back new op lower bound from 7315588116557127680 to 7315588116470304768. Current Snapshot: MvccSnapshot[applied={T|T < 7315588116557127680}]
I20260806 16:04:15.041026 20549 mvcc.cc:205] Tried to move back new op lower bound from 7315588116557127680 to 7315588116470304768. Current Snapshot: MvccSnapshot[applied={T|T < 7315588116557127680}]
I20260806 16:04:15.049386 19832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:15.049597 19832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:15.052531 19832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a21b36e5276f4c1583810efdb027b558: Sending DeleteTablet for 3 replicas of tablet b0f692474f754f4e84fcb27db4fa6c6c
I20260806 16:04:15.053515 20223 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f692474f754f4e84fcb27db4fa6c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:42248
I20260806 16:04:15.053596 20411 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f692474f754f4e84fcb27db4fa6c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52520
I20260806 16:04:15.053633 20006 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f692474f754f4e84fcb27db4fa6c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:44400
I20260806 16:04:15.054214 20555 tablet_replica.cc:330] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: stopping tablet replica
I20260806 16:04:15.054324 20556 tablet_replica.cc:330] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648: stopping tablet replica
I20260806 16:04:15.054644 20555 raft_consensus.cc:2229] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:15.054729 20556 raft_consensus.cc:2229] T b0f692474f754f4e84fcb27db4fa6c6c P 6e58b5c6226d4e128a456d7a79e56648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:15.054947 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 19921
I20260806 16:04:15.057307 20555 raft_consensus.cc:2258] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:15.059887 20557 tablet_replica.cc:330] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: stopping tablet replica
I20260806 16:04:15.060264 20557 raft_consensus.cc:2229] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:15.060838 20557 raft_consensus.cc:2258] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:15.063552 20557 ts_tablet_manager.cc:1905] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:15.064215 20555 ts_tablet_manager.cc:1905] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:15.067131 20555 ts_tablet_manager.cc:1918] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:15.067376 20555 log.cc:1196] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/b0f692474f754f4e84fcb27db4fa6c6c
I20260806 16:04:15.067507 20557 ts_tablet_manager.cc:1918] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:15.067766 20557 log.cc:1196] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/b0f692474f754f4e84fcb27db4fa6c6c
I20260806 16:04:15.067849 20555 ts_tablet_manager.cc:1939] T b0f692474f754f4e84fcb27db4fa6c6c P d6cfe7c0a86346af974c6becb8578b28: Deleting consensus metadata
I20260806 16:04:15.068209 20557 ts_tablet_manager.cc:1939] T b0f692474f754f4e84fcb27db4fa6c6c P 0602592c9e324b48b927f22510dcc6f4: Deleting consensus metadata
I20260806 16:04:15.069397 19813 catalog_manager.cc:4841] TS d6cfe7c0a86346af974c6becb8578b28 (127.13.148.2:41339): tablet b0f692474f754f4e84fcb27db4fa6c6c (table test-table [id=2aba36bab85648e39b6596c5557b6667]) successfully deleted
I20260806 16:04:15.069463 19812 catalog_manager.cc:4841] TS 0602592c9e324b48b927f22510dcc6f4 (127.13.148.3:41047): tablet b0f692474f754f4e84fcb27db4fa6c6c (table test-table [id=2aba36bab85648e39b6596c5557b6667]) successfully deleted
W20260806 16:04:15.070504 19812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:42131 (error 108)
I20260806 16:04:15.070648 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 20125
I20260806 16:04:15.073830 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.073367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42131 (local address 127.0.0.1:44414)
0806 16:04:15.073594 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:15.073711 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260806 16:04:15.074167 19812 catalog_manager.cc:4584] TS 6e58b5c6226d4e128a456d7a79e56648 (127.13.148.1:42131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0f692474f754f4e84fcb27db4fa6c6c: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42131: connect: Connection refused (error 111)
I20260806 16:04:15.080233 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 20326
I20260806 16:04:15.093185 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 19789
2026-08-06T16:04:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2628 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T16:04:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:15Z Disabled control of system clock
I20260806 16:04:15.133245 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35459
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 16:04:15.248996 20563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:15.249294 20563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:15.249336 20563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:15.253330 20563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:15.253388 20563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:15.253404 20563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:15.253422 20563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:15.253436 20563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:15.258152 20563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:15.259379 20563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:15.260742 20563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:15.269564 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:15.269614 20569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:15.269821 20563 server_base.cc:1034] running on GCE node
W20260806 16:04:15.270011 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:15.270704 20563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:15.271838 20563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:15.273044 20563 hybrid_clock.cc:648] HybridClock initialized: now 1786032255273029 us; error 29 us; skew 500 ppm
I20260806 16:04:15.275146 20563 webserver.cc:460] Webserver started at http://127.13.148.62:45501/ using document root <none> and password file <none>
I20260806 16:04:15.275810 20563 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:15.275919 20563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:15.276183 20563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:15.278410 20563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "24ce8074b56e453fab91b0c0e95c5bf4"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "b4cb84aa68317bdfe527de9c34ecfd79"
server_key_iv: "a7646c378c52c84ec0591cf98023a6bc"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.279098 20563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "24ce8074b56e453fab91b0c0e95c5bf4"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "b4cb84aa68317bdfe527de9c34ecfd79"
server_key_iv: "a7646c378c52c84ec0591cf98023a6bc"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.283102 20563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:15.285622 20583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.287201 20563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:15.287392 20563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "24ce8074b56e453fab91b0c0e95c5bf4"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "b4cb84aa68317bdfe527de9c34ecfd79"
server_key_iv: "a7646c378c52c84ec0591cf98023a6bc"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.287528 20563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:15.292688 20563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:15.293361 20563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:15.293576 20563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:15.301474 20563 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35459
I20260806 16:04:15.301499 20662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35459 every 8 connection(s)
I20260806 16:04:15.302512 20563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:15.305855 20663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:15.310111 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 20563
I20260806 16:04:15.310258 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:15.310581 13904 external_mini_cluster.cc:1428] Setting key 9ee1ae80421b51f5cf0df4b61ec6d753
I20260806 16:04:15.312057 20663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4: Bootstrap starting.
I20260806 16:04:15.314438 20663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:15.315338 20663 log.cc:824] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:15.316542 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.312021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.13.148.62:35459)
0806 16:04:15.312698 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.312718 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.312753 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:04:15.313455 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.313486 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.313883 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.314226 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:15.315109 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:15.315114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:15.315135 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:15.315147 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:15.315209 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:15.315226 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:15.315347 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.315448 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.315697 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":140,"threads_started":1}
I20260806 16:04:15.317427 20663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4: No bootstrap required, opened a new log
I20260806 16:04:15.321244 20663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ce8074b56e453fab91b0c0e95c5bf4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35459 } }
I20260806 16:04:15.321488 20663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:15.321570 20663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ce8074b56e453fab91b0c0e95c5bf4, State: Initialized, Role: FOLLOWER
I20260806 16:04:15.322000 20663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ce8074b56e453fab91b0c0e95c5bf4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35459 } }
I20260806 16:04:15.322150 20663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:15.322240 20663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:15.322367 20663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:15.323410 20663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ce8074b56e453fab91b0c0e95c5bf4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35459 } }
I20260806 16:04:15.323752 20663 leader_election.cc:302] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24ce8074b56e453fab91b0c0e95c5bf4; no voters:
I20260806 16:04:15.324079 20663 leader_election.cc:288] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:15.324162 20669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:15.324429 20669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 24ce8074b56e453fab91b0c0e95c5bf4, State: Running, Role: LEADER
I20260806 16:04:15.324788 20669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ce8074b56e453fab91b0c0e95c5bf4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35459 } }
I20260806 16:04:15.325340 20663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:15.326563 20676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ce8074b56e453fab91b0c0e95c5bf4. Latest consensus state: current_term: 1 leader_uuid: "24ce8074b56e453fab91b0c0e95c5bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ce8074b56e453fab91b0c0e95c5bf4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35459 } } }
I20260806 16:04:15.326707 20676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:15.326989 20674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24ce8074b56e453fab91b0c0e95c5bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ce8074b56e453fab91b0c0e95c5bf4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35459 } } }
I20260806 16:04:15.327185 20674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:15.327090 20689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:15.330080 20689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:15.336045 20689 catalog_manager.cc:1347] Generated new cluster ID: c693f5a1dcce424d953d9fe6d7686bf2
I20260806 16:04:15.336117 20689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:15.349812 20689 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:15.350188 20689 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:15.358485 20689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4: Generated new TSK 0
I20260806 16:04:15.359402 20689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:04:15.364202 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35459
--builtin_ntp_servers=127.13.148.20:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 16:04:15.481531 20705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:15.481851 20705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:15.481930 20705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:15.486124 20705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:15.486233 20705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:15.486335 20705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:15.492226 20705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:15.493481 20705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:15.494979 20705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:15.505832 20714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:15.505941 20713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:15.505832 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:15.506088 20705 server_base.cc:1034] running on GCE node
I20260806 16:04:15.506664 20705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:15.507366 20705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:15.508560 20705 hybrid_clock.cc:648] HybridClock initialized: now 1786032255508499 us; error 70 us; skew 500 ppm
I20260806 16:04:15.510637 20705 webserver.cc:460] Webserver started at http://127.13.148.1:45497/ using document root <none> and password file <none>
I20260806 16:04:15.511241 20705 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:15.511309 20705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:15.511528 20705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:15.513688 20705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "f52c79fdebb94acab7054d83e0812540"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "fab9a5014f08106845208db785038b06"
server_key_iv: "bc22ab925bff4762c25fdf51dec7145e"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.514348 20705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "f52c79fdebb94acab7054d83e0812540"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "fab9a5014f08106845208db785038b06"
server_key_iv: "bc22ab925bff4762c25fdf51dec7145e"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.518160 20705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 16:04:15.520869 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.522217 20705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:04:15.522378 20705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "f52c79fdebb94acab7054d83e0812540"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "fab9a5014f08106845208db785038b06"
server_key_iv: "bc22ab925bff4762c25fdf51dec7145e"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.522535 20705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:15.527572 20705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:15.528219 20705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:15.528462 20705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:15.529048 20705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:15.530000 20705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:15.530074 20705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.530145 20705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:15.530191 20705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.539642 20705 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:42737
I20260806 16:04:15.539675 20895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:42737 every 8 connection(s)
I20260806 16:04:15.540584 20705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:15.542459 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 20705
I20260806 16:04:15.542609 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:15.542927 13904 external_mini_cluster.cc:1428] Setting key d0938f2b65223a426f0aa79daf29a12c
I20260806 16:04:15.545313 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35459
--builtin_ntp_servers=127.13.148.20:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 16:04:15.547887 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.542534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38951 (local address 127.13.148.62:35459)
0806 16:04:15.543425 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.543430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.543945 (+ 515us) server_negotiation.cc:407] Connection header received
0806 16:04:15.545863 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.545866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.545901 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.545975 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:15.547349 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:15.547355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:15.547360 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:15.547378 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:15.547407 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:15.547422 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:15.547521 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.547665 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.547715 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794}
I20260806 16:04:15.548703 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.542994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35459 (local address 127.13.148.1:38951)
0806 16:04:15.543808 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:15.543851 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:15.545574 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:15.546158 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:15.546169 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:15.546283 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:15.546941 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:15.546945 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:15.547005 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:15.547103 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:15.547518 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:15.547525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:15.547530 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:04:15.547678 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 16:04:15.547871 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260806 16:04:15.550071 20896 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35459
I20260806 16:04:15.550323 20896 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:15.550850 20896 heartbeater.cc:511] Master 127.13.148.62:35459 requested a full tablet report, sending...
I20260806 16:04:15.552608 20613 ts_manager.cc:194] Registered new tserver with Master: f52c79fdebb94acab7054d83e0812540 (127.13.148.1:42737)
W20260806 16:04:15.661336 20901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:15.661863 20901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:15.661944 20901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:15.666373 20901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:15.666435 20901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:15.666563 20901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:15.674022 20901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:15.675300 20901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:15.676844 20901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:04:15.686981 20901 server_base.cc:1034] running on GCE node
W20260806 16:04:15.687114 20908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:15.687067 20911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:15.687073 20909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:15.687759 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:15.688421 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:15.689651 20901 hybrid_clock.cc:648] HybridClock initialized: now 1786032255689631 us; error 29 us; skew 500 ppm
I20260806 16:04:15.691769 20901 webserver.cc:460] Webserver started at http://127.13.148.2:44241/ using document root <none> and password file <none>
I20260806 16:04:15.692433 20901 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:15.692534 20901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:15.692785 20901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:15.695063 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "2476948dac924b2e87e5176f24e70483"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "9070286357f617fac789461974a699f1"
server_key_iv: "e71abe0e711bac611e9c8e0eec6ec2a0"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.695676 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "2476948dac924b2e87e5176f24e70483"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "9070286357f617fac789461974a699f1"
server_key_iv: "e71abe0e711bac611e9c8e0eec6ec2a0"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.699798 20901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 16:04:15.702706 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.704169 20901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:04:15.704330 20901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "2476948dac924b2e87e5176f24e70483"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "9070286357f617fac789461974a699f1"
server_key_iv: "e71abe0e711bac611e9c8e0eec6ec2a0"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.704474 20901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:15.709645 20901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:15.710306 20901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:15.710558 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:15.711161 20901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:15.712158 20901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:15.712234 20901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.712306 20901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:15.712357 20901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.722198 20901 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:43391
I20260806 16:04:15.722298 21077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:43391 every 8 connection(s)
I20260806 16:04:15.723290 20901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:15.723793 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 20901
I20260806 16:04:15.723961 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:15.724254 13904 external_mini_cluster.cc:1428] Setting key ba5a02497ddc3dd0eda36c335e8cb3db
I20260806 16:04:15.726950 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35459
--builtin_ntp_servers=127.13.148.20:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 16:04:15.730804 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.725418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:53439 (local address 127.13.148.62:35459)
0806 16:04:15.725556 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.725561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.727189 (+ 1628us) server_negotiation.cc:407] Connection header received
0806 16:04:15.728830 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.728836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.728893 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.728984 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:15.730240 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:15.730244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:15.730246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:15.730260 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:15.730284 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:15.730296 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:15.730404 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.730610 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.730662 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:04:15.731714 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.726029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35459 (local address 127.13.148.2:53439)
0806 16:04:15.727037 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:15.727084 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:15.728527 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:15.729187 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:15.729195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:15.729288 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:15.729950 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:15.729952 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:15.729996 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:15.730070 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:15.730387 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:15.730396 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:15.730402 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 16:04:15.730654 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 16:04:15.730910 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":161,"threads_started":1}
I20260806 16:04:15.732964 21078 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35459
I20260806 16:04:15.733273 21078 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:15.733821 21078 heartbeater.cc:511] Master 127.13.148.62:35459 requested a full tablet report, sending...
I20260806 16:04:15.735170 20613 ts_manager.cc:194] Registered new tserver with Master: 2476948dac924b2e87e5176f24e70483 (127.13.148.2:43391)
W20260806 16:04:15.844586 21084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:15.844877 21084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:15.844918 21084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:15.849182 21084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:15.849253 21084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:15.849310 21084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:15.854183 21084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:15.855590 21084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:15.857125 21084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:15.867260 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:15.867381 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:15.867329 21093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:15.867951 21084 server_base.cc:1034] running on GCE node
I20260806 16:04:15.868403 21084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:15.869094 21084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:15.870266 21084 hybrid_clock.cc:648] HybridClock initialized: now 1786032255870245 us; error 31 us; skew 500 ppm
I20260806 16:04:15.872330 21084 webserver.cc:460] Webserver started at http://127.13.148.3:46771/ using document root <none> and password file <none>
I20260806 16:04:15.872990 21084 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:15.873059 21084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:15.873310 21084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:15.875543 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "d832fcd804ac4a9f9e785e6c3040d91a"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "f5f12ccaa22b9b7accdd2c4634d1834d"
server_key_iv: "43b6a2086f04bb88bfac5fa629993bda"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.876154 21084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "d832fcd804ac4a9f9e785e6c3040d91a"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "f5f12ccaa22b9b7accdd2c4634d1834d"
server_key_iv: "43b6a2086f04bb88bfac5fa629993bda"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.880134 21084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:15.882994 21104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.884346 21084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:15.884500 21084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "d832fcd804ac4a9f9e785e6c3040d91a"
format_stamp: "Formatted at 2026-08-06 16:04:15 on dist-test-slave-7kzw"
server_key: "f5f12ccaa22b9b7accdd2c4634d1834d"
server_key_iv: "43b6a2086f04bb88bfac5fa629993bda"
server_key_version: "encryptionkey@0"
I20260806 16:04:15.884621 21084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:15.889439 21084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:15.890093 21084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:15.890288 21084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:15.891026 21084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:15.892191 21084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:15.892247 21084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.892323 21084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:15.892362 21084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:15.902364 21084 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:40687
I20260806 16:04:15.902405 21266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:40687 every 8 connection(s)
I20260806 16:04:15.903491 21084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:15.905344 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 21084
I20260806 16:04:15.905501 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:15.905880 13904 external_mini_cluster.cc:1428] Setting key dfdb06e08801b150e6f7066c1efba967
I20260806 16:04:15.911079 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.905625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:40307 (local address 127.13.148.62:35459)
0806 16:04:15.905825 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.905830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.907313 (+ 1483us) server_negotiation.cc:407] Connection header received
0806 16:04:15.909241 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.909244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.909284 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.909404 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:15.910610 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:15.910613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:15.910615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:15.910628 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:15.910652 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:15.910662 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:15.910753 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.910892 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.910935 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 16:04:15.911952 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.906169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35459 (local address 127.13.148.3:40307)
0806 16:04:15.907139 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:15.907188 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:15.908990 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:15.909542 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:15.909549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:15.909656 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:15.910212 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:15.910214 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:15.910256 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:15.910350 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:15.910752 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:15.910759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:15.910765 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 16:04:15.910909 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 16:04:15.911127 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":152,"threads_started":1}
I20260806 16:04:15.913213 21267 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35459
I20260806 16:04:15.913440 21267 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:15.913829 21267 heartbeater.cc:511] Master 127.13.148.62:35459 requested a full tablet report, sending...
I20260806 16:04:15.915059 20613 ts_manager.cc:194] Registered new tserver with Master: d832fcd804ac4a9f9e785e6c3040d91a (127.13.148.3:40687)
I20260806 16:04:15.920315 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:04:15.924223 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.922552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.13.148.62:35459)
0806 16:04:15.922693 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.922703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.922893 (+ 190us) server_negotiation.cc:407] Connection header received
0806 16:04:15.922978 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.922980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.923015 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.923079 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:15.923838 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:15.923841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:15.923843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:15.923856 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:15.923885 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:15.923893 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:15.923967 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.924068 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.924109 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:04:15.927773 20613 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:15.931056 20613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:15.943866 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.940974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.13.148.2:43391)
0806 16:04:15.941634 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.941644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.941663 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:04:15.941709 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.941713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.941851 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.942392 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:15.943194 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:15.943200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:15.943213 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:15.943225 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:15.943273 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:15.943306 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:15.943463 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.943641 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.943720 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":138,"threads_started":1}
I20260806 16:04:15.943853 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.940543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42737 (local address 127.0.0.1:45996)
0806 16:04:15.941299 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:15.941312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:15.941387 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:15.941997 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:15.942003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:15.942083 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:15.942896 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:15.942898 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:15.942919 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:15.942991 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:15.943538 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:15.943541 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:15.943544 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 16:04:15.943653 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 16:04:15.943688 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":102,"threads_started":1}
I20260806 16:04:15.943893 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.940174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:43391 (local address 127.0.0.1:46190)
0806 16:04:15.940845 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:15.940870 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:15.940959 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:15.941997 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:15.942003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:15.942083 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:15.943003 (+ 920us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:15.943005 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:15.943018 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:15.943055 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:15.943453 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:15.943464 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:15.943472 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:04:15.943607 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 16:04:15.943686 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":148,"threads_started":1}
I20260806 16:04:15.944453 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.941099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.13.148.1:42737)
0806 16:04:15.941549 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.941562 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.941586 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:04:15.941638 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.941645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.941821 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.942393 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:15.943368 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:15.943376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:15.943388 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:15.943402 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:15.943438 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:15.943463 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:15.943559 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.944232 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.944296 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":109,"threads_started":1}
I20260806 16:04:15.944487 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.940889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.13.148.3:40687)
0806 16:04:15.941441 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:15.941455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:15.941483 (+ 28us) server_negotiation.cc:407] Connection header received
0806 16:04:15.941543 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:15.941550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:15.941777 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:15.941934 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:15.943395 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:15.943430 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:15.943445 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:15.943459 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:15.943508 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:15.943551 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:15.943661 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:15.943925 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 16:04:15.944312 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":193,"threads_started":1}
I20260806 16:04:15.944685 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:15.940117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:40687 (local address 127.0.0.1:35150)
0806 16:04:15.940490 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:15.940543 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:15.940648 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:15.942659 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:15.942663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:15.942682 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:15.943223 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:15.943225 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:15.943240 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:15.943258 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:15.943720 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:15.943724 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:15.943728 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:04:15.943808 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 16:04:15.943839 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":111,"threads_started":1}
I20260806 16:04:15.947290 20988 tablet_service.cc:1467] Processing CreateTablet for tablet c07038da0be4455f9c63fb4dd2a54b3f (DEFAULT_TABLE table=test-table [id=39138f91d6c34939a1153ec3f8e79603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:15.947579 20803 tablet_service.cc:1467] Processing CreateTablet for tablet c07038da0be4455f9c63fb4dd2a54b3f (DEFAULT_TABLE table=test-table [id=39138f91d6c34939a1153ec3f8e79603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:15.947579 21174 tablet_service.cc:1467] Processing CreateTablet for tablet c07038da0be4455f9c63fb4dd2a54b3f (DEFAULT_TABLE table=test-table [id=39138f91d6c34939a1153ec3f8e79603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:15.948478 20988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07038da0be4455f9c63fb4dd2a54b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:15.948674 21174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07038da0be4455f9c63fb4dd2a54b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:15.948663 20803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07038da0be4455f9c63fb4dd2a54b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:15.954980 21286 tablet_bootstrap.cc:492] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Bootstrap starting.
I20260806 16:04:15.956938 21290 tablet_bootstrap.cc:492] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: Bootstrap starting.
I20260806 16:04:15.957988 21286 tablet_bootstrap.cc:654] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:15.958846 21289 tablet_bootstrap.cc:492] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540: Bootstrap starting.
I20260806 16:04:15.958916 21290 tablet_bootstrap.cc:654] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:15.958881 21286 log.cc:824] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:15.959769 21290 log.cc:824] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:15.960988 21286 tablet_bootstrap.cc:492] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: No bootstrap required, opened a new log
I20260806 16:04:15.961018 21289 tablet_bootstrap.cc:654] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:15.961221 21286 ts_tablet_manager.cc:1397] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:04:15.961844 21289 log.cc:824] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:15.961946 21290 tablet_bootstrap.cc:492] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: No bootstrap required, opened a new log
I20260806 16:04:15.962122 21290 ts_tablet_manager.cc:1397] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:04:15.965173 21286 raft_consensus.cc:348] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:15.965437 21286 raft_consensus.cc:374] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:15.965512 21286 raft_consensus.cc:729] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2476948dac924b2e87e5176f24e70483, State: Initialized, Role: FOLLOWER
I20260806 16:04:15.965745 21290 raft_consensus.cc:348] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:15.966043 21290 raft_consensus.cc:374] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:15.966105 21290 raft_consensus.cc:729] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d832fcd804ac4a9f9e785e6c3040d91a, State: Initialized, Role: FOLLOWER
I20260806 16:04:15.966068 21286 consensus_queue.cc:260] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:15.966574 21290 consensus_queue.cc:260] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:15.966831 21078 heartbeater.cc:503] Master 127.13.148.62:35459 was elected leader, sending a full tablet report...
I20260806 16:04:15.967238 21286 ts_tablet_manager.cc:1428] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:04:15.967511 21289 tablet_bootstrap.cc:492] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540: No bootstrap required, opened a new log
I20260806 16:04:15.967600 21290 ts_tablet_manager.cc:1428] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 16:04:15.967708 21289 ts_tablet_manager.cc:1397] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260806 16:04:15.967885 21267 heartbeater.cc:503] Master 127.13.148.62:35459 was elected leader, sending a full tablet report...
I20260806 16:04:15.972329 21289 raft_consensus.cc:348] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:15.972543 21289 raft_consensus.cc:374] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:15.972616 21289 raft_consensus.cc:729] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52c79fdebb94acab7054d83e0812540, State: Initialized, Role: FOLLOWER
I20260806 16:04:15.973052 21289 consensus_queue.cc:260] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:15.973758 20896 heartbeater.cc:503] Master 127.13.148.62:35459 was elected leader, sending a full tablet report...
I20260806 16:04:15.973966 21289 ts_tablet_manager.cc:1428] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260806 16:04:15.974323 21079 tablet.cc:2369] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:04:16.041931 20897 tablet.cc:2369] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:04:16.099728 21295 raft_consensus.cc:482] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:16.100018 21295 raft_consensus.cc:504] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:16.101223 21295 leader_election.cc:288] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d832fcd804ac4a9f9e785e6c3040d91a (127.13.148.3:40687), f52c79fdebb94acab7054d83e0812540 (127.13.148.1:42737)
I20260806 16:04:16.103438 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.101543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:40687 (local address 127.13.148.2:44951)
0806 16:04:16.101692 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:16.101709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:16.101818 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:16.102155 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:16.102158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:16.102173 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:16.102794 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:16.102799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:16.102814 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:16.102845 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:16.103137 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:16.103143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:16.103147 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:04:16.103239 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 16:04:16.103282 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 16:04:16.103470 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.101697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:44951 (local address 127.13.148.3:40687)
0806 16:04:16.101847 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:16.101854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:16.101872 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:04:16.101927 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:16.101930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:16.101962 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:16.102034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:16.102958 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:16.102963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:16.102966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:16.102981 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:16.103007 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:16.103020 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:16.103141 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:16.103234 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 16:04:16.103296 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:04:16.104058 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.101543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42737 (local address 127.13.148.2:53575)
0806 16:04:16.102426 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:16.102438 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:16.102595 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:16.102894 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:16.102900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:16.102917 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:16.103425 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:16.103427 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:16.103437 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:16.103464 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:16.103818 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:16.103821 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:16.103823 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 16:04:16.103899 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 16:04:16.103930 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":534,"thread_start_us":136,"threads_started":1}
I20260806 16:04:16.104097 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.101681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:53575 (local address 127.13.148.1:42737)
0806 16:04:16.101833 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:16.101838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:16.102585 (+ 747us) server_negotiation.cc:407] Connection header received
0806 16:04:16.102710 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:16.102713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:16.102758 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:16.102841 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:16.103619 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:16.103624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:16.103628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:16.103643 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:16.103670 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:16.103682 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:16.103746 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:16.103933 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 16:04:16.103983 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:04:16.104241 21197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07038da0be4455f9c63fb4dd2a54b3f" candidate_uuid: "2476948dac924b2e87e5176f24e70483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" is_pre_election: true
I20260806 16:04:16.104624 21197 raft_consensus.cc:2454] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2476948dac924b2e87e5176f24e70483 in term 0.
I20260806 16:04:16.104830 20830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07038da0be4455f9c63fb4dd2a54b3f" candidate_uuid: "2476948dac924b2e87e5176f24e70483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52c79fdebb94acab7054d83e0812540" is_pre_election: true
I20260806 16:04:16.105116 20830 raft_consensus.cc:2454] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2476948dac924b2e87e5176f24e70483 in term 0.
I20260806 16:04:16.105201 20925 leader_election.cc:302] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2476948dac924b2e87e5176f24e70483, d832fcd804ac4a9f9e785e6c3040d91a; no voters:
I20260806 16:04:16.105502 21295 raft_consensus.cc:2781] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:16.105619 21295 raft_consensus.cc:482] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:16.105680 21295 raft_consensus.cc:3037] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:16.106850 21295 raft_consensus.cc:504] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:16.107211 21295 leader_election.cc:288] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [CANDIDATE]: Term 1 election: Requested vote from peers d832fcd804ac4a9f9e785e6c3040d91a (127.13.148.3:40687), f52c79fdebb94acab7054d83e0812540 (127.13.148.1:42737)
I20260806 16:04:16.107712 21197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07038da0be4455f9c63fb4dd2a54b3f" candidate_uuid: "2476948dac924b2e87e5176f24e70483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d832fcd804ac4a9f9e785e6c3040d91a"
I20260806 16:04:16.107837 20830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07038da0be4455f9c63fb4dd2a54b3f" candidate_uuid: "2476948dac924b2e87e5176f24e70483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52c79fdebb94acab7054d83e0812540"
I20260806 16:04:16.107877 21197 raft_consensus.cc:3037] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:16.107955 20830 raft_consensus.cc:3037] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:16.109306 21197 raft_consensus.cc:2454] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2476948dac924b2e87e5176f24e70483 in term 1.
I20260806 16:04:16.109306 20830 raft_consensus.cc:2454] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2476948dac924b2e87e5176f24e70483 in term 1.
I20260806 16:04:16.109664 20924 leader_election.cc:302] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2476948dac924b2e87e5176f24e70483, f52c79fdebb94acab7054d83e0812540; no voters:
I20260806 16:04:16.109891 21295 raft_consensus.cc:2781] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:16.110193 21295 raft_consensus.cc:686] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 1 LEADER]: Becoming Leader. State: Replica: 2476948dac924b2e87e5176f24e70483, State: Running, Role: LEADER
I20260806 16:04:16.110723 21295 consensus_queue.cc:237] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } }
I20260806 16:04:16.114341 20608 catalog_manager.cc:5495] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2476948dac924b2e87e5176f24e70483 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "2476948dac924b2e87e5176f24e70483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2476948dac924b2e87e5176f24e70483" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 } health_report { overall_health: UNKNOWN } } }
I20260806 16:04:16.134585 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.133000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.13.148.2:43391)
0806 16:04:16.133153 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:16.133157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:16.133170 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:04:16.133248 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:16.133250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:16.133309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:16.133382 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:16.134157 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:16.134160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:16.134162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:16.134173 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:16.134192 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:16.134200 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:16.134283 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:16.134328 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 16:04:16.134361 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 16:04:16.140697 21197 raft_consensus.cc:1261] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 1 FOLLOWER]: Refusing update from remote peer 2476948dac924b2e87e5176f24e70483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:16.140697 20830 raft_consensus.cc:1261] T c07038da0be4455f9c63fb4dd2a54b3f P f52c79fdebb94acab7054d83e0812540 [term 1 FOLLOWER]: Refusing update from remote peer 2476948dac924b2e87e5176f24e70483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:16.141570 21295 consensus_queue.cc:1035] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d832fcd804ac4a9f9e785e6c3040d91a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:16.141862 21300 consensus_queue.cc:1035] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52c79fdebb94acab7054d83e0812540" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:16.150414 21304 mvcc.cc:205] Tried to move back new op lower bound from 7315588121144344576 to 7315588121032617984. Current Snapshot: MvccSnapshot[applied={T|T < 7315588121144344576}]
I20260806 16:04:16.159368 20608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:16.159591 20608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:16.162282 20608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24ce8074b56e453fab91b0c0e95c5bf4: Sending DeleteTablet for 3 replicas of tablet c07038da0be4455f9c63fb4dd2a54b3f
I20260806 16:04:16.163208 20988 tablet_service.cc:1514] Processing DeleteTablet for tablet c07038da0be4455f9c63fb4dd2a54b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:16 UTC) from {username='slave'} at 127.0.0.1:46190
I20260806 16:04:16.163224 21174 tablet_service.cc:1514] Processing DeleteTablet for tablet c07038da0be4455f9c63fb4dd2a54b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:16 UTC) from {username='slave'} at 127.0.0.1:35150
I20260806 16:04:16.163367 20803 tablet_service.cc:1514] Processing DeleteTablet for tablet c07038da0be4455f9c63fb4dd2a54b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:16 UTC) from {username='slave'} at 127.0.0.1:45996
I20260806 16:04:16.163792 21318 tablet_replica.cc:330] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: stopping tablet replica
I20260806 16:04:16.164088 21319 tablet_replica.cc:330] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: stopping tablet replica
I20260806 16:04:16.164425 21319 raft_consensus.cc:2229] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:16.164574 21318 raft_consensus.cc:2229] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:16.164613 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 20705
I20260806 16:04:16.164860 21319 raft_consensus.cc:2258] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:16.164896 21318 raft_consensus.cc:2258] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:16.168336 21318 ts_tablet_manager.cc:1905] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:16.171241 21318 ts_tablet_manager.cc:1918] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:16.171340 21318 log.cc:1196] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/c07038da0be4455f9c63fb4dd2a54b3f
I20260806 16:04:16.171658 21318 ts_tablet_manager.cc:1939] T c07038da0be4455f9c63fb4dd2a54b3f P d832fcd804ac4a9f9e785e6c3040d91a: Deleting consensus metadata
I20260806 16:04:16.172981 20594 catalog_manager.cc:4841] TS d832fcd804ac4a9f9e785e6c3040d91a (127.13.148.3:40687): tablet c07038da0be4455f9c63fb4dd2a54b3f (table test-table [id=39138f91d6c34939a1153ec3f8e79603]) successfully deleted
I20260806 16:04:16.176054 21319 ts_tablet_manager.cc:1905] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:16.178956 21319 ts_tablet_manager.cc:1918] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:16.179070 21319 log.cc:1196] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/c07038da0be4455f9c63fb4dd2a54b3f
I20260806 16:04:16.179407 21319 ts_tablet_manager.cc:1939] T c07038da0be4455f9c63fb4dd2a54b3f P 2476948dac924b2e87e5176f24e70483: Deleting consensus metadata
I20260806 16:04:16.180795 20584 catalog_manager.cc:4841] TS 2476948dac924b2e87e5176f24e70483 (127.13.148.2:43391): tablet c07038da0be4455f9c63fb4dd2a54b3f (table test-table [id=39138f91d6c34939a1153ec3f8e79603]) successfully deleted
W20260806 16:04:16.184664 20593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:42737 (error 108)
I20260806 16:04:16.184811 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 20901
I20260806 16:04:16.190543 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.190094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42737 (local address 127.0.0.1:46010)
0806 16:04:16.190273 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:16.190374 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 16:04:16.191076 20593 catalog_manager.cc:4584] TS f52c79fdebb94acab7054d83e0812540 (127.13.148.1:42737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c07038da0be4455f9c63fb4dd2a54b3f: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42737: connect: Connection refused (error 111)
I20260806 16:04:16.196645 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 21084
I20260806 16:04:16.207363 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 20563
2026-08-06T16:04:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1112 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 16:04:16.236068 13904 mini_oidc.cc:152] Starting JWKS server
I20260806 16:04:16.237816 13904 webserver.cc:460] Webserver started at https://127.0.0.1:43381/ using document root <none> and password file <none>
I20260806 16:04:16.238005 13904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 16:04:16.238328 13904 webserver.cc:460] Webserver started at http://127.0.0.1:45157/ using document root <none> and password file <none>
2026-08-06T16:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:16Z Disabled control of system clock
I20260806 16:04:16.249518 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34859
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:35777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:34859
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43381/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/testchainca.cert with env {}
W20260806 16:04:16.364027 21335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:16.364329 21335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:16.364372 21335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:16.368501 21335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:16.368579 21335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:16.368602 21335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:16.368618 21335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:16.368634 21335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 16:04:16.368650 21335 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 16:04:16.368716 21335 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43381/jwks.json
I20260806 16:04:16.374294 21335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:34859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43381/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:16.375538 21335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:16.376672 21335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:16.385491 21340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:16.385546 21341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:16.385511 21343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:16.385738 21335 server_base.cc:1034] running on GCE node
I20260806 16:04:16.386361 21335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:16.387519 21335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:16.388752 21335 hybrid_clock.cc:648] HybridClock initialized: now 1786032256388736 us; error 28 us; skew 500 ppm
I20260806 16:04:16.390862 21335 webserver.cc:460] Webserver started at http://127.13.148.62:42875/ using document root <none> and password file <none>
I20260806 16:04:16.391480 21335 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:16.391590 21335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:16.391840 21335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:16.393962 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "03e82c81f2164345ba7ecefcd10fc81b"
format_stamp: "Formatted at 2026-08-06 16:04:16 on dist-test-slave-7kzw"
server_key: "6ca91e4266bdcafc9e8ac085ce1d0f55"
server_key_iv: "be32f3fbe20da6899882763d1590f394"
server_key_version: "encryptionkey@0"
I20260806 16:04:16.394635 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "03e82c81f2164345ba7ecefcd10fc81b"
format_stamp: "Formatted at 2026-08-06 16:04:16 on dist-test-slave-7kzw"
server_key: "6ca91e4266bdcafc9e8ac085ce1d0f55"
server_key_iv: "be32f3fbe20da6899882763d1590f394"
server_key_version: "encryptionkey@0"
I20260806 16:04:16.398572 21335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:04:16.401156 21349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:16.402575 21335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 16:04:16.402737 21335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "03e82c81f2164345ba7ecefcd10fc81b"
format_stamp: "Formatted at 2026-08-06 16:04:16 on dist-test-slave-7kzw"
server_key: "6ca91e4266bdcafc9e8ac085ce1d0f55"
server_key_iv: "be32f3fbe20da6899882763d1590f394"
server_key_version: "encryptionkey@0"
I20260806 16:04:16.402866 21335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:16.418614 21335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:16.419381 21335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:16.419638 21335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:16.427152 21335 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:34859
I20260806 16:04:16.427181 21424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:34859 every 8 connection(s)
I20260806 16:04:16.428205 21335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:16.431792 21425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:16.436450 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 21335
I20260806 16:04:16.436628 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:16.436893 13904 external_mini_cluster.cc:1428] Setting key 468334684c97e0d6b4a0eaafe437257f
I20260806 16:04:16.437764 21425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b: Bootstrap starting.
I20260806 16:04:16.440703 21425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:16.441641 21425 log.cc:824] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:16.443960 21425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b: No bootstrap required, opened a new log
I20260806 16:04:16.447655 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.438567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.13.148.62:34859)
0806 16:04:16.439396 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:16.439457 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:16.439503 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:04:16.440372 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:16.440399 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:16.440925 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:16.441309 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:16.441777 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.443208 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.444201 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.444716 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.444907 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:16.445804 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:16.445810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:16.445823 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:16.446217 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:16.446276 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:16.446297 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:16.446412 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:16.446589 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 16:04:16.446856 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":306,"thread_start_us":121,"threads_started":1}
I20260806 16:04:16.447734 21425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e82c81f2164345ba7ecefcd10fc81b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34859 } }
I20260806 16:04:16.448064 21425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:16.448132 21425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e82c81f2164345ba7ecefcd10fc81b, State: Initialized, Role: FOLLOWER
I20260806 16:04:16.448559 21425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e82c81f2164345ba7ecefcd10fc81b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34859 } }
I20260806 16:04:16.448714 21425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:16.448781 21425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:16.448910 21425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:16.450002 21425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e82c81f2164345ba7ecefcd10fc81b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34859 } }
I20260806 16:04:16.450392 21425 leader_election.cc:302] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03e82c81f2164345ba7ecefcd10fc81b; no voters:
I20260806 16:04:16.450765 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:04:16.450773 21425 leader_election.cc:288] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:16.450843 21430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:16.451052 21430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [term 1 LEADER]: Becoming Leader. State: Replica: 03e82c81f2164345ba7ecefcd10fc81b, State: Running, Role: LEADER
I20260806 16:04:16.451359 21430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e82c81f2164345ba7ecefcd10fc81b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34859 } }
I20260806 16:04:16.452070 21425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:16.453047 21434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03e82c81f2164345ba7ecefcd10fc81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e82c81f2164345ba7ecefcd10fc81b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34859 } } }
I20260806 16:04:16.453197 21434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:16.453567 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e82c81f2164345ba7ecefcd10fc81b. Latest consensus state: current_term: 1 leader_uuid: "03e82c81f2164345ba7ecefcd10fc81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e82c81f2164345ba7ecefcd10fc81b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34859 } } }
I20260806 16:04:16.453616 21443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:16.453675 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:16.456856 21443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:16.463044 21443 catalog_manager.cc:1347] Generated new cluster ID: 9a23c6afc8c84bed8b8b112fcdf7de74
I20260806 16:04:16.463132 21443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:16.475180 21443 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:16.476435 21443 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:16.483810 21443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e82c81f2164345ba7ecefcd10fc81b: Generated new TSK 0
I20260806 16:04:16.484654 21443 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 16:04:16.514679 21463 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 16:04:16.514891 21463 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 16:04:16.515590 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.494874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.13.148.62:34859)
0806 16:04:16.495049 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:16.495055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:16.495253 (+ 198us) server_negotiation.cc:407] Connection header received
0806 16:04:16.495519 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:16.495523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:16.495591 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:16.495718 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0806 16:04:16.496021 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.496660 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.497638 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.497916 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.498036 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:16.498451 (+ 415us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 16:04:16.515038 (+ 16587us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 16:04:16.515409 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 16:04:16.515712 21428 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 16:04:16.516997 21462 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:16.494736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:34859 (local address 127.0.0.1:60126)
0806 16:04:16.495142 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:16.495164 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:16.495336 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:16.495739 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:16.495743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:16.495760 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0806 16:04:16.495895 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:16.495901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.496791 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.496796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:16.497497 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:16.497505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.498058 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.498063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:16.498176 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:16.498224 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 16:04:16.515326 (+ 17102us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 16:04:16.515598 (+ 272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.62:34859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.148.62:34859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 16:04:16.517936 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 21335
2026-08-06T16:04:16Z chronyd exiting
I20260806 16:04:16.562000 13904 test_util.cc:175] -----------------------------------------------
I20260806 16:04:16.562135 13904 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786032227601881-13904-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 16:04:16.563481 13904 mini_oidc.cc:152] Starting JWKS server
I20260806 16:04:16.564459 13904 webserver.cc:460] Webserver started at https://127.0.0.1:42901/ using document root <none> and password file <none>
I20260806 16:04:16.564608 13904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 16:04:16.564941 13904 webserver.cc:460] Webserver started at http://127.0.0.1:45025/ using document root <none> and password file <none>
2026-08-06T16:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:16Z Disabled control of system clock
I20260806 16:04:16.575964 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45175
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:45175
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42901/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/testchainca.cert with env {}
W20260806 16:04:16.692688 21470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:16.692976 21470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:16.693015 21470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:16.697048 21470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:16.697115 21470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:16.697131 21470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:16.697149 21470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:16.697163 21470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 16:04:16.697180 21470 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 16:04:16.697193 21470 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42901/jwks.json
I20260806 16:04:16.702623 21470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:45175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42901/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:16.703663 21470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:16.704638 21470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:16.713763 21487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:16.713788 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:16.713788 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:16.713970 21470 server_base.cc:1034] running on GCE node
I20260806 16:04:16.714730 21470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:16.715934 21470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:16.717123 21470 hybrid_clock.cc:648] HybridClock initialized: now 1786032256717087 us; error 48 us; skew 500 ppm
I20260806 16:04:16.719304 21470 webserver.cc:460] Webserver started at http://127.13.148.62:44077/ using document root <none> and password file <none>
I20260806 16:04:16.719928 21470 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:16.719995 21470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:16.720165 21470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:16.722277 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "d564cd9be13145e083b4a246eb619549"
format_stamp: "Formatted at 2026-08-06 16:04:16 on dist-test-slave-7kzw"
server_key: "4642d61e426ad425094eb29f4812bbaa"
server_key_iv: "2b65517959bc47adec073c3d307d0907"
server_key_version: "encryptionkey@0"
I20260806 16:04:16.723107 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "d564cd9be13145e083b4a246eb619549"
format_stamp: "Formatted at 2026-08-06 16:04:16 on dist-test-slave-7kzw"
server_key: "4642d61e426ad425094eb29f4812bbaa"
server_key_iv: "2b65517959bc47adec073c3d307d0907"
server_key_version: "encryptionkey@0"
I20260806 16:04:16.727221 21470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:16.729825 21495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:16.731312 21470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:16.731426 21470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "d564cd9be13145e083b4a246eb619549"
format_stamp: "Formatted at 2026-08-06 16:04:16 on dist-test-slave-7kzw"
server_key: "4642d61e426ad425094eb29f4812bbaa"
server_key_iv: "2b65517959bc47adec073c3d307d0907"
server_key_version: "encryptionkey@0"
I20260806 16:04:16.731521 21470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:16.746251 21470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:16.747117 21470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:16.747324 21470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:16.755954 21470 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:45175
I20260806 16:04:16.756064 21567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:45175 every 8 connection(s)
I20260806 16:04:16.757122 21470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:16.761204 21568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:16.762703 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 21470
I20260806 16:04:16.762806 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:16.763137 13904 external_mini_cluster.cc:1428] Setting key 6c68fc346840fe0f236498b562389180
I20260806 16:04:16.768599 21568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549: Bootstrap starting.
I20260806 16:04:16.771448 21568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:16.772367 21568 log.cc:824] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:16.773348 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.764736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.13.148.62:45175)
0806 16:04:16.765622 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:16.765640 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:16.765679 (+ 39us) server_negotiation.cc:407] Connection header received
0806 16:04:16.766325 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:16.766342 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:16.766834 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:16.767197 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:16.767678 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.768631 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.769617 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.770093 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.770270 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:16.771323 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:16.771329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:16.771342 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:16.771785 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:16.771844 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:16.771866 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:16.771981 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:16.772146 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 16:04:16.772421 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":179,"threads_started":1}
I20260806 16:04:16.775040 21568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549: No bootstrap required, opened a new log
I20260806 16:04:16.776746 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:04:16.779668 21568 raft_consensus.cc:348] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d564cd9be13145e083b4a246eb619549" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45175 } }
I20260806 16:04:16.779963 21568 raft_consensus.cc:374] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:16.780062 21568 raft_consensus.cc:729] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d564cd9be13145e083b4a246eb619549, State: Initialized, Role: FOLLOWER
I20260806 16:04:16.780618 21568 consensus_queue.cc:260] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d564cd9be13145e083b4a246eb619549" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45175 } }
I20260806 16:04:16.780804 21568 raft_consensus.cc:388] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:16.780917 21568 raft_consensus.cc:482] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:16.781054 21568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:16.782198 21568 raft_consensus.cc:504] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d564cd9be13145e083b4a246eb619549" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45175 } }
I20260806 16:04:16.782778 21568 leader_election.cc:302] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d564cd9be13145e083b4a246eb619549; no voters:
I20260806 16:04:16.783233 21568 leader_election.cc:288] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:16.783366 21578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:16.783690 21578 raft_consensus.cc:686] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [term 1 LEADER]: Becoming Leader. State: Replica: d564cd9be13145e083b4a246eb619549, State: Running, Role: LEADER
I20260806 16:04:16.784041 21578 consensus_queue.cc:237] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d564cd9be13145e083b4a246eb619549" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45175 } }
I20260806 16:04:16.784962 21568 sys_catalog.cc:564] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:16.785687 21585 sys_catalog.cc:455] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d564cd9be13145e083b4a246eb619549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d564cd9be13145e083b4a246eb619549" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45175 } } }
I20260806 16:04:16.785822 21585 sys_catalog.cc:458] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:16.786214 21592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:16.787653 21587 sys_catalog.cc:455] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d564cd9be13145e083b4a246eb619549. Latest consensus state: current_term: 1 leader_uuid: "d564cd9be13145e083b4a246eb619549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d564cd9be13145e083b4a246eb619549" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45175 } } }
I20260806 16:04:16.787768 21587 sys_catalog.cc:458] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:16.789700 21592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:16.795862 21592 catalog_manager.cc:1347] Generated new cluster ID: 3fde8ead09ee490880c16f712af5044a
I20260806 16:04:16.795953 21592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:16.812734 21592 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:16.813845 21592 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:16.823969 21592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d564cd9be13145e083b4a246eb619549: Generated new TSK 0
I20260806 16:04:16.824793 21592 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 16:04:16.864327 21619 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 16:04:16.864437 21619 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 16:04:16.865224 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:16.841592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.13.148.62:45175)
0806 16:04:16.841745 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:16.841749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:16.841908 (+ 159us) server_negotiation.cc:407] Connection header received
0806 16:04:16.842083 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:16.842085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:16.842134 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:16.842227 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0806 16:04:16.842529 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.843068 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.844345 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.844571 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.844690 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:16.845045 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 16:04:16.864798 (+ 19753us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 16:04:16.865077 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34092: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 16:04:16.865336 21575 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34092: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 16:04:16.865430 21618 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:16.841511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45175 (local address 127.0.0.1:34092)
0806 16:04:16.841828 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:16.841843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:16.841965 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:16.842262 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:16.842265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:16.842273 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 16:04:16.842390 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:16.842396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.843187 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.843192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:16.844197 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:16.844209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:16.844731 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:16.844735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:16.844835 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:16.844864 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 16:04:16.865037 (+ 20173us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 16:04:16.865252 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.62:45175: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42901/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":124,"threads_started":1}
I20260806 16:04:16.866416 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 21470
2026-08-06T16:04:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (352 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 16:04:16.915620 13904 mini_oidc.cc:152] Starting JWKS server
I20260806 16:04:16.916534 13904 webserver.cc:460] Webserver started at https://127.0.0.1:45503/ using document root <none> and password file <none>
I20260806 16:04:16.916671 13904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 16:04:16.916941 13904 webserver.cc:460] Webserver started at http://127.0.0.1:45789/ using document root <none> and password file <none>
2026-08-06T16:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:16Z Disabled control of system clock
I20260806 16:04:16.928651 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39303
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39303
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45503/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 16:04:17.052590 21629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:17.052911 21629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:17.052954 21629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:17.056972 21629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:17.057037 21629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:17.057052 21629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:17.057070 21629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:17.057085 21629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 16:04:17.057101 21629 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 16:04:17.057114 21629 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45503/jwks.json
I20260806 16:04:17.062141 21629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45503/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:17.063464 21629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:17.064970 21629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:17.074673 21637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:17.074908 21629 server_base.cc:1034] running on GCE node
W20260806 16:04:17.074729 21641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:17.074705 21639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:17.075767 21629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:17.076990 21629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:17.078183 21629 hybrid_clock.cc:648] HybridClock initialized: now 1786032257078147 us; error 48 us; skew 500 ppm
I20260806 16:04:17.080389 21629 webserver.cc:460] Webserver started at http://127.13.148.62:40389/ using document root <none> and password file <none>
I20260806 16:04:17.081037 21629 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:17.081106 21629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:17.081339 21629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:17.083645 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "fa212a3994d1470ebb98731128f3f3fa"
format_stamp: "Formatted at 2026-08-06 16:04:17 on dist-test-slave-7kzw"
server_key: "1d4bd6d5d11b864547194b5e271cf5b0"
server_key_iv: "905c6c3e3cf1069b8df1e4db53139936"
server_key_version: "encryptionkey@0"
I20260806 16:04:17.084271 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "fa212a3994d1470ebb98731128f3f3fa"
format_stamp: "Formatted at 2026-08-06 16:04:17 on dist-test-slave-7kzw"
server_key: "1d4bd6d5d11b864547194b5e271cf5b0"
server_key_iv: "905c6c3e3cf1069b8df1e4db53139936"
server_key_version: "encryptionkey@0"
I20260806 16:04:17.088356 21629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 16:04:17.090883 21649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:17.092195 21629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:04:17.092312 21629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "fa212a3994d1470ebb98731128f3f3fa"
format_stamp: "Formatted at 2026-08-06 16:04:17 on dist-test-slave-7kzw"
server_key: "1d4bd6d5d11b864547194b5e271cf5b0"
server_key_iv: "905c6c3e3cf1069b8df1e4db53139936"
server_key_version: "encryptionkey@0"
I20260806 16:04:17.092434 21629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:17.108079 21629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:17.108855 21629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:17.109078 21629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:17.116508 21629 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39303
I20260806 16:04:17.116532 21720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39303 every 8 connection(s)
I20260806 16:04:17.117600 21629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:17.121071 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:17.126447 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 21629
I20260806 16:04:17.126680 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:17.126940 13904 external_mini_cluster.cc:1428] Setting key 3761fcfffb31ac6f6d3361740d36df9a
I20260806 16:04:17.127147 21722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa: Bootstrap starting.
I20260806 16:04:17.129704 21722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:17.130540 21722 log.cc:824] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:17.132791 21722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa: No bootstrap required, opened a new log
I20260806 16:04:17.136541 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:17.128429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.13.148.62:39303)
0806 16:04:17.129094 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:17.129108 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:17.129145 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:04:17.129791 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:17.129809 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:17.130258 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:17.130613 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:17.131018 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.131966 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.133147 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.133625 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.133761 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:17.134715 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:17.134720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:17.134734 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:17.135123 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:17.135185 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:17.135206 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:17.135285 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:17.135465 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 16:04:17.135805 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":161,"threads_started":1}
I20260806 16:04:17.136996 21722 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa212a3994d1470ebb98731128f3f3fa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39303 } }
I20260806 16:04:17.137251 21722 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:17.137336 21722 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa212a3994d1470ebb98731128f3f3fa, State: Initialized, Role: FOLLOWER
I20260806 16:04:17.137853 21722 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa212a3994d1470ebb98731128f3f3fa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39303 } }
I20260806 16:04:17.138019 21722 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:17.138085 21722 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:17.138181 21722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:17.139410 21722 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa212a3994d1470ebb98731128f3f3fa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39303 } }
I20260806 16:04:17.139804 21722 leader_election.cc:302] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa212a3994d1470ebb98731128f3f3fa; no voters:
I20260806 16:04:17.140113 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:04:17.140183 21722 leader_election.cc:288] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:17.140269 21732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:17.140455 21732 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [term 1 LEADER]: Becoming Leader. State: Replica: fa212a3994d1470ebb98731128f3f3fa, State: Running, Role: LEADER
I20260806 16:04:17.140762 21732 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa212a3994d1470ebb98731128f3f3fa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39303 } }
I20260806 16:04:17.141551 21722 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:17.142750 21735 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa212a3994d1470ebb98731128f3f3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa212a3994d1470ebb98731128f3f3fa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39303 } } }
I20260806 16:04:17.142915 21735 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:17.143313 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa212a3994d1470ebb98731128f3f3fa. Latest consensus state: current_term: 1 leader_uuid: "fa212a3994d1470ebb98731128f3f3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa212a3994d1470ebb98731128f3f3fa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39303 } } }
I20260806 16:04:17.143409 21747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:17.143424 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:17.147598 21747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:17.153927 21747 catalog_manager.cc:1347] Generated new cluster ID: 3076ac3c69534e1c8f8ba6a742133941
I20260806 16:04:17.154024 21747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:17.165387 21747 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:17.166785 21747 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:17.172420 21747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa212a3994d1470ebb98731128f3f3fa: Generated new TSK 0
I20260806 16:04:17.173306 21747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:04:17.189994 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:17.184332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58654 (local address 127.13.148.62:39303)
0806 16:04:17.184506 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:17.184511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:17.184681 (+ 170us) server_negotiation.cc:407] Connection header received
0806 16:04:17.184883 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:17.184886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:17.184942 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:17.185036 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0806 16:04:17.185321 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.185901 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.187215 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.187461 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.187555 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:17.187979 (+ 424us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 16:04:17.189571 (+ 1592us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 16:04:17.189832 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 16:04:17.190078 21728 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 16:04:17.191828 21767 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:17.184231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39303 (local address 127.0.0.1:58654)
0806 16:04:17.184570 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:17.184586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:17.184722 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:17.185074 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:17.185077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:17.185085 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 16:04:17.185208 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:17.185214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.186018 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.186022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:17.187064 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:17.187076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.187604 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.187608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:17.187702 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:17.187733 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 16:04:17.189767 (+ 2034us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 16:04:17.191640 (+ 1873us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.62:39303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45503/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":1653,"thread_start_us":126,"threads_started":1}
I20260806 16:04:17.192430 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 21629
E20260806 16:04:17.204927 21768 webserver.cc:533] Webserver: sslize failed:
E20260806 16:04:17.205058 21768 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T16:04:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T16:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:17Z Disabled control of system clock
I20260806 16:04:17.253682 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46211
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:42149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46211
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/testchainca.cert with env {}
W20260806 16:04:17.374739 21772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:17.375054 21772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:17.375097 21772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:17.379268 21772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:17.379344 21772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:17.379360 21772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:17.379379 21772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:17.379393 21772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 16:04:17.379410 21772 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 16:04:17.379423 21772 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 16:04:17.385069 21772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:17.386148 21772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:17.387233 21772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:17.396101 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:17.396086 21782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:17.396132 21779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:17.396227 21772 server_base.cc:1034] running on GCE node
I20260806 16:04:17.397078 21772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:17.398164 21772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:17.399367 21772 hybrid_clock.cc:648] HybridClock initialized: now 1786032257399344 us; error 35 us; skew 500 ppm
I20260806 16:04:17.401387 21772 webserver.cc:460] Webserver started at http://127.13.148.62:35149/ using document root <none> and password file <none>
I20260806 16:04:17.401997 21772 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:17.402094 21772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:17.402335 21772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:17.404444 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "85b4b01eb0f1452bb89357835ea43d54"
format_stamp: "Formatted at 2026-08-06 16:04:17 on dist-test-slave-7kzw"
server_key: "3d94437e710e4c660d63224fadd8351f"
server_key_iv: "ae63fbea38bab64d88e2095b5137aa30"
server_key_version: "encryptionkey@0"
I20260806 16:04:17.405036 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "85b4b01eb0f1452bb89357835ea43d54"
format_stamp: "Formatted at 2026-08-06 16:04:17 on dist-test-slave-7kzw"
server_key: "3d94437e710e4c660d63224fadd8351f"
server_key_iv: "ae63fbea38bab64d88e2095b5137aa30"
server_key_version: "encryptionkey@0"
I20260806 16:04:17.408952 21772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:04:17.411737 21790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:17.413055 21772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:04:17.413223 21772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "85b4b01eb0f1452bb89357835ea43d54"
format_stamp: "Formatted at 2026-08-06 16:04:17 on dist-test-slave-7kzw"
server_key: "3d94437e710e4c660d63224fadd8351f"
server_key_iv: "ae63fbea38bab64d88e2095b5137aa30"
server_key_version: "encryptionkey@0"
I20260806 16:04:17.413349 21772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:17.434345 21772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:17.435159 21772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:17.435384 21772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:17.442616 21867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46211 every 8 connection(s)
I20260806 16:04:17.442631 21772 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46211
I20260806 16:04:17.443657 21772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:17.446899 21868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:17.451292 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 21772
I20260806 16:04:17.451431 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:17.451728 13904 external_mini_cluster.cc:1428] Setting key 17be69545b24664c2749086587f21f35
I20260806 16:04:17.453511 21868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54: Bootstrap starting.
I20260806 16:04:17.456590 21868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:17.457660 21868 log.cc:824] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:17.459977 21868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54: No bootstrap required, opened a new log
I20260806 16:04:17.463312 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:17.453245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.13.148.62:46211)
0806 16:04:17.453995 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:17.454017 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:17.454063 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:04:17.455058 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:17.455088 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:17.455655 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:17.456059 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:17.456580 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.458264 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.459376 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.459918 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.460123 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:17.461259 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:17.461264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:17.461279 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:17.461725 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:17.461793 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:17.461816 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:17.461946 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:17.462125 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 16:04:17.462419 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260806 16:04:17.464287 21868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b4b01eb0f1452bb89357835ea43d54" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46211 } }
I20260806 16:04:17.464607 21868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:17.464704 21868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b4b01eb0f1452bb89357835ea43d54, State: Initialized, Role: FOLLOWER
I20260806 16:04:17.465209 21868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b4b01eb0f1452bb89357835ea43d54" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46211 } }
I20260806 16:04:17.465376 21868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:17.465458 21868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:17.465551 21868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:17.466702 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:04:17.466758 21868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b4b01eb0f1452bb89357835ea43d54" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46211 } }
I20260806 16:04:17.467145 21868 leader_election.cc:302] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85b4b01eb0f1452bb89357835ea43d54; no voters:
I20260806 16:04:17.467514 21868 leader_election.cc:288] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:17.467657 21874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:17.467936 21874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [term 1 LEADER]: Becoming Leader. State: Replica: 85b4b01eb0f1452bb89357835ea43d54, State: Running, Role: LEADER
I20260806 16:04:17.468282 21874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b4b01eb0f1452bb89357835ea43d54" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46211 } }
I20260806 16:04:17.468874 21868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:17.470142 21877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b4b01eb0f1452bb89357835ea43d54. Latest consensus state: current_term: 1 leader_uuid: "85b4b01eb0f1452bb89357835ea43d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b4b01eb0f1452bb89357835ea43d54" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46211 } } }
I20260806 16:04:17.470299 21877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:17.470652 21875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85b4b01eb0f1452bb89357835ea43d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b4b01eb0f1452bb89357835ea43d54" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46211 } } }
I20260806 16:04:17.470786 21875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:17.470868 21889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:17.474227 21889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:17.480551 21889 catalog_manager.cc:1347] Generated new cluster ID: 1e2cc0f8be6f4dbb94b6db0bfe6626b1
I20260806 16:04:17.480635 21889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:17.507696 21889 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:17.508911 21889 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:17.515045 21889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85b4b01eb0f1452bb89357835ea43d54: Generated new TSK 0
I20260806 16:04:17.515949 21889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:04:17.572760 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:17.533641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.13.148.62:46211)
0806 16:04:17.533798 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:17.533803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:17.533937 (+ 134us) server_negotiation.cc:407] Connection header received
0806 16:04:17.534113 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:17.534117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:17.534180 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:17.534313 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0806 16:04:17.534766 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.535723 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.536749 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.537157 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.537274 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:17.537605 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 16:04:17.572228 (+ 34623us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 16:04:17.572584 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52334: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 16:04:17.572868 21872 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52334: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 16:04:17.572919 21909 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:17.533533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46211 (local address 127.0.0.1:52334)
0806 16:04:17.533835 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:17.533848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:17.533964 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:17.534329 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:17.534332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:17.534340 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 16:04:17.534517 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:17.534530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.535891 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.535895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:17.536578 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:17.536587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:17.537297 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:17.537300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:17.537373 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:17.537399 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 16:04:17.572489 (+ 35090us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 16:04:17.572712 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.62:46211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
I20260806 16:04:17.573594 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 21772
2026-08-06T16:04:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (357 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (134 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:17 dist-test-slave-7kzw krb5kdc[21927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:17 dist-test-slave-7kzw krb5kdc[21927](info): set up 2 sockets
Aug 06 16:04:17 dist-test-slave-7kzw krb5kdc[21927](info): commencing operation
krb5kdc: starting...
W20260806 16:04:19.126206 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:19 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:19.267378 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46815
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:19.387315 21944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:19.387598 21944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:19.387640 21944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:19.391762 21944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:19.391824 21944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:19.391844 21944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:19.391860 21944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:19.391875 21944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:19.397255 21944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:19.398375 21944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:19.399473 21944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:19.408102 21954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:19.408094 21951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:19.408094 21952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:19.408360 21944 server_base.cc:1034] running on GCE node
I20260806 16:04:19.408985 21944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:19.410135 21944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:19.411334 21944 hybrid_clock.cc:648] HybridClock initialized: now 1786032259411316 us; error 32 us; skew 500 ppm
Aug 06 16:04:19 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032259, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:19.418514 21944 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:19.419483 21944 webserver.cc:460] Webserver started at http://127.13.148.62:34527/ using document root <none> and password file <none>
I20260806 16:04:19.420047 21944 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:19.420110 21944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:19.420281 21944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:19.422371 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "274aaedf510746c1a68fddbf99501304"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "cb264ccf12c8bc56addfee07da8af280"
server_key_iv: "e9a595c5d72f80109aa6588b71b0da8a"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.422986 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "274aaedf510746c1a68fddbf99501304"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "cb264ccf12c8bc56addfee07da8af280"
server_key_iv: "e9a595c5d72f80109aa6588b71b0da8a"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.427021 21944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:04:19.429584 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:19.430977 21944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:19.431097 21944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "274aaedf510746c1a68fddbf99501304"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "cb264ccf12c8bc56addfee07da8af280"
server_key_iv: "e9a595c5d72f80109aa6588b71b0da8a"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.431188 21944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:19.445569 21944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:19.449260 21944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:19.449437 21944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:19.457268 21944 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46815
I20260806 16:04:19.457297 22041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46815 every 8 connection(s)
I20260806 16:04:19.458269 21944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:19.461225 22042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:19.465757 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 21944
I20260806 16:04:19.465945 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:19.466225 13904 external_mini_cluster.cc:1428] Setting key e10c66e538e2967c87f5c42df0a0d8aa
I20260806 16:04:19.467092 22042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304: Bootstrap starting.
I20260806 16:04:19.469628 22042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:19.470402 22042 log.cc:824] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:19.472397 22042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304: No bootstrap required, opened a new log
Aug 06 16:04:19 dist-test-slave-7kzw krb5kdc[21927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:19.476325 22042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274aaedf510746c1a68fddbf99501304" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46815 } }
I20260806 16:04:19.476660 22042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:19.476744 22042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274aaedf510746c1a68fddbf99501304, State: Initialized, Role: FOLLOWER
I20260806 16:04:19.477262 22042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274aaedf510746c1a68fddbf99501304" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46815 } }
I20260806 16:04:19.477421 22042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:19.477512 22042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:19.477618 22042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:19.478615 22042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274aaedf510746c1a68fddbf99501304" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46815 } }
I20260806 16:04:19.478955 22042 leader_election.cc:302] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 274aaedf510746c1a68fddbf99501304; no voters:
I20260806 16:04:19.479333 22042 leader_election.cc:288] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:19.479408 22049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:19.479652 22049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [term 1 LEADER]: Becoming Leader. State: Replica: 274aaedf510746c1a68fddbf99501304, State: Running, Role: LEADER
I20260806 16:04:19.480001 22049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274aaedf510746c1a68fddbf99501304" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46815 } }
I20260806 16:04:19.480685 22042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:19.481997 22051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "274aaedf510746c1a68fddbf99501304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274aaedf510746c1a68fddbf99501304" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46815 } } }
I20260806 16:04:19.482156 22051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:19.482211 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:19.467783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.13.148.62:46815)
0806 16:04:19.468255 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:19.468271 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:19.468307 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:04:19.468971 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:19.468989 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:19.469326 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:19.469679 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:19.470553 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:19.471764 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:19.472691 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:19.473172 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:19.473382 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:19.476085 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:19.476090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:19.476104 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:19.476133 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:19.478368 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:19.478973 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:19.478980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:19.478987 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:19.479193 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:19.479602 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:19.479608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:19.479611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:19.480177 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:19.480403 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:19.480746 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 16:04:19.481091 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":125,"threads_started":1}
I20260806 16:04:19.482767 22053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 274aaedf510746c1a68fddbf99501304. Latest consensus state: current_term: 1 leader_uuid: "274aaedf510746c1a68fddbf99501304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274aaedf510746c1a68fddbf99501304" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46815 } } }
I20260806 16:04:19.482942 22053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304 [sys.catalog]: This master's current role is: LEADER
W20260806 16:04:19.485832 22069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:04:19.485925 22069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:04:19.486004 22061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:19.486946 22061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:19.492982 22061 catalog_manager.cc:1347] Generated new cluster ID: f4c119ec830f48e09386941de29536f8
I20260806 16:04:19.493059 22061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:19.499106 22061 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:19.500069 22061 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:19.507146 22061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 274aaedf510746c1a68fddbf99501304: Generated new TSK 0
I20260806 16:04:19.507937 22061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:04:19.554929 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46815
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:19.701146 22081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:19.701478 22081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:19.701572 22081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:19.706068 22081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:19.706183 22081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:19.706282 22081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:19.711686 22081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:19.712857 22081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:19.713933 22081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:19.723834 22090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:19.723886 22092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:19.723834 22089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:19.724303 22081 server_base.cc:1034] running on GCE node
I20260806 16:04:19.724774 22081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:19.725440 22081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:19.726634 22081 hybrid_clock.cc:648] HybridClock initialized: now 1786032259726611 us; error 31 us; skew 500 ppm
Aug 06 16:04:19 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032259, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:19.729619 22081 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:04:19.730527 22081 webserver.cc:460] Webserver started at http://127.13.148.1:36081/ using document root <none> and password file <none>
I20260806 16:04:19.731076 22081 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:19.731137 22081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:19.731307 22081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:19.733302 22081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "b279c024460040499a77e63dea3bf648"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "69b8bf874158e49a286217cf4f2f11d6"
server_key_iv: "4bad128a4b78982f76d151cd811ef197"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.733837 22081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "b279c024460040499a77e63dea3bf648"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "69b8bf874158e49a286217cf4f2f11d6"
server_key_iv: "4bad128a4b78982f76d151cd811ef197"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.737779 22081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 16:04:19.740520 22102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:19.741868 22081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 16:04:19.742038 22081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "b279c024460040499a77e63dea3bf648"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "69b8bf874158e49a286217cf4f2f11d6"
server_key_iv: "4bad128a4b78982f76d151cd811ef197"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.742172 22081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:19.760874 22081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:19.764396 22081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:19.764621 22081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:19.765262 22081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:19.766147 22081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:19.766220 22081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:19.766296 22081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:19.766348 22081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:19.775590 22081 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:36977
I20260806 16:04:19.775632 22272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:36977 every 8 connection(s)
I20260806 16:04:19.776679 22081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:19.784991 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 22081
I20260806 16:04:19.785099 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:19.785458 13904 external_mini_cluster.cc:1428] Setting key 439295ad6b72ceb002483de565053bfc
Aug 06 16:04:19 dist-test-slave-7kzw krb5kdc[21927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032259, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:19.793437 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:19.778641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38501 (local address 127.13.148.62:46815)
0806 16:04:19.778819 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:19.778823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:19.779675 (+ 852us) server_negotiation.cc:407] Connection header received
0806 16:04:19.781451 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:19.781454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:19.781506 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:19.781591 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:19.783256 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:19.783818 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:19.785375 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:19.785718 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:19.785846 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:19.789369 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:19.789377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:19.789383 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:19.789425 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:19.791172 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:19.791779 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:19.791782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:19.791783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:19.791868 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:19.792249 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:19.792253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:19.792255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:19.792459 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:19.792573 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:19.793120 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 16:04:19.793262 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 16:04:19.794377 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:19.778971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46815 (local address 127.13.148.1:38501)
0806 16:04:19.779505 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:19.779553 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:19.781197 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:19.781875 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:19.781885 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:19.782386 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:19.783041 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:19.783075 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:19.783982 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:19.783985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:19.785235 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:19.785247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:19.785942 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:19.785948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:19.786190 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:19.787026 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:19.787045 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:19.789162 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:19.791314 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:19.791320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:19.791335 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:19.791647 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:19.791985 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:19.791988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:19.791989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:19.792134 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:19.792574 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:19.792582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:19.792863 (+ 281us) client_negotiation.cc:769] Sending connection context
0806 16:04:19.793060 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 16:04:19.793434 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":125,"threads_started":1}
I20260806 16:04:19.795750 22273 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46815
I20260806 16:04:19.796022 22273 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:19.796542 22273 heartbeater.cc:511] Master 127.13.148.62:46815 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:04:19.798413 21987 ts_manager.cc:194] Registered new tserver with Master: b279c024460040499a77e63dea3bf648 (127.13.148.1:36977)
I20260806 16:04:19.799796 21987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:38501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:04:19.821966 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46815
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:19.938002 22283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:19.938297 22283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:19.938344 22283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:19.942555 22283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:19.942626 22283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:19.942718 22283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:19.948278 22283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:19.949429 22283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:19.950593 22283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:19.961002 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:19.961007 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:19.961241 22290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:19.961313 22283 server_base.cc:1034] running on GCE node
I20260806 16:04:19.961841 22283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:19.962558 22283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:19.963761 22283 hybrid_clock.cc:648] HybridClock initialized: now 1786032259963739 us; error 33 us; skew 500 ppm
Aug 06 16:04:19 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032259, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:19.966766 22283 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:04:19.967720 22283 webserver.cc:460] Webserver started at http://127.13.148.2:39975/ using document root <none> and password file <none>
I20260806 16:04:19.968255 22283 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:19.968317 22283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:19.968487 22283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:19.970578 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "682c1a211b8d4d2db21388eca72bbc64"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "538dc640d8d0bc892197c5ae1ab90e89"
server_key_iv: "e5b954d9e504471ab91599d6f985a58b"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.971135 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "682c1a211b8d4d2db21388eca72bbc64"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "538dc640d8d0bc892197c5ae1ab90e89"
server_key_iv: "e5b954d9e504471ab91599d6f985a58b"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.974962 22283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 16:04:19.977461 22309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:19.978963 22283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:04:19.979079 22283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "682c1a211b8d4d2db21388eca72bbc64"
format_stamp: "Formatted at 2026-08-06 16:04:19 on dist-test-slave-7kzw"
server_key: "538dc640d8d0bc892197c5ae1ab90e89"
server_key_iv: "e5b954d9e504471ab91599d6f985a58b"
server_key_version: "encryptionkey@0"
I20260806 16:04:19.979183 22283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:19.997979 22283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:20.001206 22283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:20.001400 22283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:20.001957 22283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:20.002874 22283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:20.002928 22283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:20.002971 22283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:20.002986 22283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:20.013069 22283 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:40983
I20260806 16:04:20.013090 22483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:40983 every 8 connection(s)
I20260806 16:04:20.014004 22283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:20.020601 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 22283
I20260806 16:04:20.020802 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:20.021095 13904 external_mini_cluster.cc:1428] Setting key 79a7ec6af2fa96a30bbdef84309324a3
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032259, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:20.029204 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:20.015947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:60051 (local address 127.13.148.62:46815)
0806 16:04:20.016096 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:20.016100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:20.016935 (+ 835us) server_negotiation.cc:407] Connection header received
0806 16:04:20.018354 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:20.018358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:20.018420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:20.018558 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:20.019772 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.020313 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.021655 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.021885 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.022002 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:20.024966 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:20.024969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:20.024972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:20.025000 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:20.026781 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.027354 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.027359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.027361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.027449 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.028010 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.028012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.028013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.028209 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:20.028343 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:20.028892 (+ 549us) server_negotiation.cc:299] Negotiation successful
0806 16:04:20.029035 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:04:20.030121 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:20.016321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46815 (local address 127.13.148.2:60051)
0806 16:04:20.016776 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:20.016820 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:20.018105 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:20.018742 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:20.018756 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:20.019128 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:20.019612 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:20.019626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.020449 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.020453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:20.021515 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:20.021527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.022048 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.022052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:20.022231 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:20.022911 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:20.022929 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:20.024770 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:20.026915 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:20.026922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:20.026939 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:20.027227 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:20.027598 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:20.027606 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:20.027611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:20.027870 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:20.028343 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:20.028352 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:20.028628 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 16:04:20.028848 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 16:04:20.029203 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260806 16:04:20.031814 22484 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46815
I20260806 16:04:20.032182 22484 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:20.032707 22484 heartbeater.cc:511] Master 127.13.148.62:46815 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:04:20.034071 21987 ts_manager.cc:194] Registered new tserver with Master: 682c1a211b8d4d2db21388eca72bbc64 (127.13.148.2:40983)
I20260806 16:04:20.034935 21987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:60051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:04:20.054132 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46815
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:20.170285 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:20.170663 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:20.170748 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:20.174826 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:20.174937 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:20.175040 22491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:20.180508 22491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:20.181782 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:20.183081 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:20.193599 22501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:20.193658 22491 server_base.cc:1034] running on GCE node
W20260806 16:04:20.193635 22503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:20.193614 22507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:20.194361 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:20.195058 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:20.196259 22491 hybrid_clock.cc:648] HybridClock initialized: now 1786032260196240 us; error 27 us; skew 500 ppm
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:20.199514 22491 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:04:20.200475 22491 webserver.cc:460] Webserver started at http://127.13.148.3:37059/ using document root <none> and password file <none>
I20260806 16:04:20.201119 22491 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:20.201220 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:20.201493 22491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:20.203699 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "6b64d1e8ce284bd084080a1c28d44207"
format_stamp: "Formatted at 2026-08-06 16:04:20 on dist-test-slave-7kzw"
server_key: "01870932c1cbc2755fa8466dfed8cee3"
server_key_iv: "f1de5f5b66658b522325433b7a835c6e"
server_key_version: "encryptionkey@0"
I20260806 16:04:20.204247 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "6b64d1e8ce284bd084080a1c28d44207"
format_stamp: "Formatted at 2026-08-06 16:04:20 on dist-test-slave-7kzw"
server_key: "01870932c1cbc2755fa8466dfed8cee3"
server_key_iv: "f1de5f5b66658b522325433b7a835c6e"
server_key_version: "encryptionkey@0"
I20260806 16:04:20.208158 22491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:20.210752 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:20.212090 22491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:04:20.212193 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "6b64d1e8ce284bd084080a1c28d44207"
format_stamp: "Formatted at 2026-08-06 16:04:20 on dist-test-slave-7kzw"
server_key: "01870932c1cbc2755fa8466dfed8cee3"
server_key_iv: "f1de5f5b66658b522325433b7a835c6e"
server_key_version: "encryptionkey@0"
I20260806 16:04:20.212280 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:20.235670 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:20.239573 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:20.239900 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:20.240665 22491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:20.241716 22491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:20.241777 22491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:20.241832 22491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:20.241902 22491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:20.251508 22491 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:44907
I20260806 16:04:20.251535 22679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:44907 every 8 connection(s)
I20260806 16:04:20.252417 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:20.262580 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 22491
I20260806 16:04:20.262728 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:20.263319 13904 external_mini_cluster.cc:1428] Setting key 2bad2318ebe1e85f75826c47d4f2e4c9
I20260806 16:04:20.268338 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:20.254340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50481 (local address 127.13.148.62:46815)
0806 16:04:20.254543 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:20.254549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:20.255347 (+ 798us) server_negotiation.cc:407] Connection header received
0806 16:04:20.256694 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:20.256697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:20.256747 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:20.256823 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:20.258261 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.259131 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.260324 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.260550 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.260645 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:20.263818 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:20.263822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:20.263824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:20.263854 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:20.265908 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.266585 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.266589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.266591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.266726 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.267095 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.267098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.267099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.267333 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:20.267420 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:20.268093 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 16:04:20.268210 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 16:04:20.269359 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:20.254692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46815 (local address 127.13.148.3:50481)
0806 16:04:20.255169 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:20.255231 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:20.256486 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:20.256989 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:20.256995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:20.257382 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:20.258074 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:20.258103 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.259277 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.259279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:20.260180 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:20.260190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.260695 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.260698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:20.260853 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:20.261530 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:20.261548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:20.263584 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:20.266046 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:20.266053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:20.266071 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:20.266430 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:20.266821 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:20.266823 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:20.266825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:20.266994 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:20.267470 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:20.267476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:20.267823 (+ 347us) client_negotiation.cc:769] Sending connection context
0806 16:04:20.268021 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 16:04:20.268354 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260806 16:04:20.270762 22681 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46815
I20260806 16:04:20.271065 22681 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:20.271631 22681 heartbeater.cc:511] Master 127.13.148.62:46815 requested a full tablet report, sending...
I20260806 16:04:20.272818 21987 ts_manager.cc:194] Registered new tserver with Master: 6b64d1e8ce284bd084080a1c28d44207 (127.13.148.3:44907)
I20260806 16:04:20.273419 21987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:50481
I20260806 16:04:20.278343 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/test-admin.keytab.
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:20.295841 13904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 16:04:20.299088 22693 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:20.297410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.13.148.1:36977)
0806 16:04:20.297736 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:20.297743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:20.297771 (+ 28us) server_negotiation.cc:407] Connection header received
0806 16:04:20.297823 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:20.297827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:20.298017 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:20.298173 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:20.298910 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36564: BlockingRecv error: recv got EOF from 127.0.0.1:36564 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":119,"threads_started":1}
W20260806 16:04:20.299451 13904 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5cc' of type FILE
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:20.300632 13904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:20.301586 13904 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:04:20.312258 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:20.302599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.13.148.1:36977)
0806 16:04:20.302737 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:20.302742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:20.302866 (+ 124us) server_negotiation.cc:407] Connection header received
0806 16:04:20.302959 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:20.302961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:20.303004 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:20.303076 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:20.303588 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.304446 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.305302 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.305708 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.305800 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:20.308379 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:20.308389 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:20.308404 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:20.308431 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:20.310100 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.310637 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.310643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.310645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.310911 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.311232 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.311235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.311236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.311506 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:20.311728 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:20.311884 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 16:04:20.312094 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 16:04:20.316723 22693 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:20.315432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.13.148.1:36977)
0806 16:04:20.315580 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:20.315585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:20.315748 (+ 163us) server_negotiation.cc:407] Connection header received
0806 16:04:20.315912 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:20.315916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:20.315961 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:20.316087 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:20.316583 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36572: BlockingRecv error: recv got EOF from 127.0.0.1:36572 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 16:04:20.316936 13904 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5cc' of type FILE
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:20.318145 13904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:20.319254 13904 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:04:20.328702 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:20.320161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.13.148.1:36977)
0806 16:04:20.320288 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:20.320292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:20.320493 (+ 201us) server_negotiation.cc:407] Connection header received
0806 16:04:20.320608 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:20.320614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:20.320665 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:20.320748 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:20.321257 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.321755 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.322665 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.322909 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.323039 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:20.325303 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:20.325307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:20.325308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:20.325331 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:20.326895 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.327381 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.327385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.327387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.327477 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.327817 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.327821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.327823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.328003 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:20.328179 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:20.328415 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 16:04:20.328534 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 16:04:20.331547 22693 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:20.330444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.13.148.1:36977)
0806 16:04:20.330606 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:20.330610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:20.330766 (+ 156us) server_negotiation.cc:407] Connection header received
0806 16:04:20.330904 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:20.330907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:20.330950 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:20.331031 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:20.331437 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36592: BlockingRecv error: recv got EOF from 127.0.0.1:36592 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260806 16:04:20.331763 13904 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786032227601881-13904-0/krb5kdc/krb5cc' of type FILE
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:20.332995 13904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:20.334008 13904 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[21927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:04:20.343757 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:20.335129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.13.148.1:36977)
0806 16:04:20.335253 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:20.335294 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:20.335314 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:04:20.335490 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:20.335498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:20.335540 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:20.335617 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:20.336165 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.336717 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.337595 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:20.337832 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:20.337921 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:20.340442 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:20.340446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:20.340448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:20.340476 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:20.341982 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.342444 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.342447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.342448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.342603 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:20.342871 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:20.342874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:20.342875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:20.343045 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:20.343176 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:20.343466 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 16:04:20.343564 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:04:20.344950 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 22081
I20260806 16:04:20.357378 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 22283
I20260806 16:04:20.379737 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 22491
I20260806 16:04:20.397166 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 21944
2026-08-06T16:04:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2558 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[22714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[22714](info): set up 2 sockets
Aug 06 16:04:20 dist-test-slave-7kzw krb5kdc[22714](info): commencing operation
krb5kdc: starting...
W20260806 16:04:21.623452 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:21 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.62.keytab.
Entry for principal oryx/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.62.keytab.
I20260806 16:04:21.771592 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44525
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.62.keytab
--principal=oryx/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:21.891860 22729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:21.892215 22729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:21.892295 22729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:21.896452 22729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:21.896569 22729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:21.896615 22729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:21.896679 22729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:21.896724 22729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:21.902137 22729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:44525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.62.keytab
--principal=oryx/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:21.903343 22729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:21.904464 22729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:21.913743 22744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:21.913829 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:21.913740 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:21.914189 22729 server_base.cc:1034] running on GCE node
I20260806 16:04:21.914919 22729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:21.916147 22729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:21.917332 22729 hybrid_clock.cc:648] HybridClock initialized: now 1786032261917303 us; error 47 us; skew 500 ppm
Aug 06 16:04:21 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032261, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:21.920730 22729 init.cc:377] Logged in from keytab as oryx/127.13.148.62@KRBTEST.COM (short username oryx)
I20260806 16:04:21.921739 22729 webserver.cc:460] Webserver started at http://127.13.148.62:35223/ using document root <none> and password file <none>
I20260806 16:04:21.922384 22729 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:21.922508 22729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:21.922782 22729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:21.925154 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "1684ee6e90fd424487d2e316d689569a"
format_stamp: "Formatted at 2026-08-06 16:04:21 on dist-test-slave-7kzw"
server_key: "01c5d230fb8178a9a4d3aaa7a2f492ea"
server_key_iv: "655116a38a03be6c81c57ebdcca1f258"
server_key_version: "encryptionkey@0"
I20260806 16:04:21.925832 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "1684ee6e90fd424487d2e316d689569a"
format_stamp: "Formatted at 2026-08-06 16:04:21 on dist-test-slave-7kzw"
server_key: "01c5d230fb8178a9a4d3aaa7a2f492ea"
server_key_iv: "655116a38a03be6c81c57ebdcca1f258"
server_key_version: "encryptionkey@0"
I20260806 16:04:21.930106 22729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 16:04:21.932864 22752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:21.934247 22729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:04:21.934417 22729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "1684ee6e90fd424487d2e316d689569a"
format_stamp: "Formatted at 2026-08-06 16:04:21 on dist-test-slave-7kzw"
server_key: "01c5d230fb8178a9a4d3aaa7a2f492ea"
server_key_iv: "655116a38a03be6c81c57ebdcca1f258"
server_key_version: "encryptionkey@0"
I20260806 16:04:21.934574 22729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:21.950137 22729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:21.953800 22729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:21.954041 22729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:21.961658 22729 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:44525
I20260806 16:04:21.961995 22829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:44525 every 8 connection(s)
I20260806 16:04:21.962742 22729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:21.965725 22831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:21.970343 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 22729
I20260806 16:04:21.970530 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:21.970844 13904 external_mini_cluster.cc:1428] Setting key 2beff81ad1ab52838ef9808d88deb8c0
I20260806 16:04:21.971547 22831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: Bootstrap starting.
I20260806 16:04:21.973843 22831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:21.974649 22831 log.cc:824] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:21.976567 22831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: No bootstrap required, opened a new log
Aug 06 16:04:21 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:21.981639 22831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } }
I20260806 16:04:21.982012 22831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:21.982077 22831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1684ee6e90fd424487d2e316d689569a, State: Initialized, Role: FOLLOWER
I20260806 16:04:21.982704 22831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } }
I20260806 16:04:21.982939 22831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:21.983064 22831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:21.983223 22831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:21.984611 22831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } }
I20260806 16:04:21.985036 22831 leader_election.cc:302] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1684ee6e90fd424487d2e316d689569a; no voters:
I20260806 16:04:21.985437 22831 leader_election.cc:288] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:21.985618 22838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:21.985864 22838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 1 LEADER]: Becoming Leader. State: Replica: 1684ee6e90fd424487d2e316d689569a, State: Running, Role: LEADER
I20260806 16:04:21.986290 22838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } }
I20260806 16:04:21.986846 22831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:21.987886 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:21.972320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.13.148.62:44525)
0806 16:04:21.972733 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:21.972750 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:21.972786 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:04:21.973415 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:21.973431 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:21.973741 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:21.974064 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:21.974961 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:21.975929 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:21.976834 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:21.977313 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:21.977491 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:21.981223 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:21.981234 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:21.981252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:21.981285 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:21.983790 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:21.984301 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:21.984310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:21.984318 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:21.984468 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:21.984836 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:21.984839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:21.984841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:21.985471 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:21.986446 (+ 975us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:21.986633 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 16:04:21.986945 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":148,"threads_started":1}
I20260806 16:04:21.990144 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } }
I20260806 16:04:21.990271 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:21.990710 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1684ee6e90fd424487d2e316d689569a. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } }
I20260806 16:04:21.990814 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:21.991559 22852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:21.995121 22852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:22.003183 22852 catalog_manager.cc:1347] Generated new cluster ID: 21ed52bc8b6644e9ac2115ac33bbbcf9
I20260806 16:04:22.003273 22852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:22.022284 22852 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:22.023690 22852 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:22.035121 22852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: Generated new TSK 0
I20260806 16:04:22.035996 22852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
I20260806 16:04:22.077752 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44525
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:22.193173 22869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:22.193421 22869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:22.193465 22869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:22.197424 22869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:22.197487 22869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:22.197544 22869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:22.202622 22869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:22.203625 22869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:22.204700 22869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:22.214525 22875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:22.214692 22877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:22.214537 22881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:22.214852 22869 server_base.cc:1034] running on GCE node
I20260806 16:04:22.215368 22869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:22.215965 22869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:22.217127 22869 hybrid_clock.cc:648] HybridClock initialized: now 1786032262217122 us; error 32 us; skew 500 ppm
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:22.220003 22869 init.cc:377] Logged in from keytab as oryx/127.13.148.1@KRBTEST.COM (short username oryx)
I20260806 16:04:22.220875 22869 webserver.cc:460] Webserver started at http://127.13.148.1:36777/ using document root <none> and password file <none>
I20260806 16:04:22.221429 22869 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:22.221491 22869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:22.221758 22869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:22.223814 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "e42cbfbe63404bd39c4850e55be882f5"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "2ea94fa1748a56309b0721cf052ae793"
server_key_iv: "a142f356a9332de96f59ddf99d6ac933"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.224367 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "e42cbfbe63404bd39c4850e55be882f5"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "2ea94fa1748a56309b0721cf052ae793"
server_key_iv: "a142f356a9332de96f59ddf99d6ac933"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.228221 22869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:04:22.230655 22890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.231958 22869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:04:22.232089 22869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "e42cbfbe63404bd39c4850e55be882f5"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "2ea94fa1748a56309b0721cf052ae793"
server_key_iv: "a142f356a9332de96f59ddf99d6ac933"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.232201 22869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:22.256633 22869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:22.260087 22869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:22.260309 22869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:22.260970 22869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:22.261837 22869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:22.261904 22869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.261950 22869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:22.262003 22869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.271700 22869 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:35813
I20260806 16:04:22.271720 23074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:35813 every 8 connection(s)
I20260806 16:04:22.272610 22869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:22.275419 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 22869
I20260806 16:04:22.275564 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:22.275914 13904 external_mini_cluster.cc:1428] Setting key 0483658b5ea07c1ab12d0be52f00cdb9
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:04:22.288692 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.274574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:56367 (local address 127.13.148.62:44525)
0806 16:04:22.274733 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:22.274738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:22.275637 (+ 899us) server_negotiation.cc:407] Connection header received
0806 16:04:22.277245 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:22.277250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:22.277302 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:22.277395 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:22.279088 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.279734 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.280912 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.281157 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.281284 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.284503 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:22.284506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:22.284508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:22.284533 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:22.286082 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.286806 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.286809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.286810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:22.286894 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.287315 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.287317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.287318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:22.287493 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:22.287566 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:22.288423 (+ 857us) server_negotiation.cc:299] Negotiation successful
0806 16:04:22.288535 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:04:22.289633 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.274964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.1:56367)
0806 16:04:22.275472 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:22.275526 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:22.276990 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:22.277647 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:22.277657 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:22.278194 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:22.278938 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:22.278962 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.279881 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.279883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:22.280781 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:22.280791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.281329 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.281333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:22.281565 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.282288 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:22.282314 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:22.284320 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:22.286217 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:22.286224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:22.286241 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:22.286684 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:22.286991 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:22.286994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:22.286997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:22.287199 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:22.287608 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:22.287617 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:22.288095 (+ 478us) client_negotiation.cc:769] Sending connection context
0806 16:04:22.288328 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 16:04:22.288696 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.2@KRBTEST.COM" created.
I20260806 16:04:22.291194 23075 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44525
I20260806 16:04:22.291451 23075 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:22.291994 23075 heartbeater.cc:511] Master 127.13.148.62:44525 requested a full tablet report, sending...
I20260806 16:04:22.293543 22774 ts_manager.cc:194] Registered new tserver with Master: e42cbfbe63404bd39c4850e55be882f5 (127.13.148.1:35813)
I20260806 16:04:22.294799 22774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:56367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
I20260806 16:04:22.309465 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44525
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:22.420485 23080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:22.420737 23080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:22.420778 23080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:22.424747 23080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:22.424808 23080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:22.424899 23080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:22.430066 23080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:22.431146 23080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:22.432237 23080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:22.442384 23088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:22.442355 23095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:22.442359 23089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:22.442881 23080 server_base.cc:1034] running on GCE node
I20260806 16:04:22.443238 23080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:22.443801 23080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:22.444996 23080 hybrid_clock.cc:648] HybridClock initialized: now 1786032262444969 us; error 31 us; skew 500 ppm
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:22.448132 23080 init.cc:377] Logged in from keytab as oryx/127.13.148.2@KRBTEST.COM (short username oryx)
I20260806 16:04:22.449043 23080 webserver.cc:460] Webserver started at http://127.13.148.2:40217/ using document root <none> and password file <none>
I20260806 16:04:22.449631 23080 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:22.449740 23080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:22.449981 23080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:22.452153 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "1cd0c89391ee4de4948cd407678ac1d0"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "88f07daebc075675d36c61b3735f1b9c"
server_key_iv: "37c65f266079bf1af9185b0bc5e643e6"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.452872 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "1cd0c89391ee4de4948cd407678ac1d0"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "88f07daebc075675d36c61b3735f1b9c"
server_key_iv: "37c65f266079bf1af9185b0bc5e643e6"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.457085 23080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:04:22.459553 23104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.460813 23080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:04:22.460952 23080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "1cd0c89391ee4de4948cd407678ac1d0"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "88f07daebc075675d36c61b3735f1b9c"
server_key_iv: "37c65f266079bf1af9185b0bc5e643e6"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.461068 23080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:22.488464 23080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:22.492147 23080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:22.492360 23080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:22.492985 23080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:22.493840 23080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:22.493916 23080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.493989 23080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:22.494040 23080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.503360 23080 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:40533
I20260806 16:04:22.503392 23274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:40533 every 8 connection(s)
I20260806 16:04:22.504324 23080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:22.506800 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 23080
I20260806 16:04:22.506942 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:22.507232 13904 external_mini_cluster.cc:1428] Setting key a2da5784962d7c5ff9464b99597531b6
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:04:22.519430 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.506231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:41801 (local address 127.13.148.62:44525)
0806 16:04:22.506368 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:22.506371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:22.507315 (+ 944us) server_negotiation.cc:407] Connection header received
0806 16:04:22.508926 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:22.508929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:22.508982 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:22.509065 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:22.510261 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.510860 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.512205 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.512517 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.512606 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.515400 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:22.515403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:22.515405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:22.515430 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:22.516951 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.517627 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.517629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.517630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:22.517736 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.518167 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.518169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.518170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:22.518342 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:22.518417 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:22.519195 (+ 778us) server_negotiation.cc:299] Negotiation successful
0806 16:04:22.519302 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:04:22.520429 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.506617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.2:41801)
0806 16:04:22.507115 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:22.507179 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:22.508718 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:22.509260 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:22.509267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:22.509632 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:22.510106 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:22.510119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.511037 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.511043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:22.512067 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:22.512077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.512671 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.512674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:22.512823 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.513464 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:22.513482 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:22.515220 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:22.517088 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:22.517095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:22.517112 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:22.517520 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:22.517838 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:22.517844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:22.517849 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:22.518053 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:22.518494 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:22.518504 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:22.518874 (+ 370us) client_negotiation.cc:769] Sending connection context
0806 16:04:22.519132 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 16:04:22.519506 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260806 16:04:22.521777 23275 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44525
I20260806 16:04:22.522017 23275 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.3@KRBTEST.COM" created.
I20260806 16:04:22.522655 23275 heartbeater.cc:511] Master 127.13.148.62:44525 requested a full tablet report, sending...
I20260806 16:04:22.523789 22774 ts_manager.cc:194] Registered new tserver with Master: 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533)
I20260806 16:04:22.524412 22774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:41801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
I20260806 16:04:22.542083 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44525
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:22.658715 23286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:22.658974 23286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:22.659018 23286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:22.663225 23286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:22.663290 23286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:22.663380 23286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:22.669027 23286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:22.670172 23286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:22.671315 23286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:22.681291 23297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:22.681243 23292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:22.681253 23299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:22.681495 23286 server_base.cc:1034] running on GCE node
I20260806 16:04:22.682011 23286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:22.682719 23286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:22.683892 23286 hybrid_clock.cc:648] HybridClock initialized: now 1786032262683870 us; error 31 us; skew 500 ppm
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:22.686720 23286 init.cc:377] Logged in from keytab as oryx/127.13.148.3@KRBTEST.COM (short username oryx)
I20260806 16:04:22.687572 23286 webserver.cc:460] Webserver started at http://127.13.148.3:43333/ using document root <none> and password file <none>
I20260806 16:04:22.688208 23286 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:22.688272 23286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:22.688510 23286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:22.690711 23286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "52f2160488f74534b5117515552567f7"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "11451bcd22223b76bffa80c56bc67ca7"
server_key_iv: "c555ac48508e00ff9454fcbeb89e5686"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.691273 23286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "52f2160488f74534b5117515552567f7"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "11451bcd22223b76bffa80c56bc67ca7"
server_key_iv: "c555ac48508e00ff9454fcbeb89e5686"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.695107 23286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:22.697650 23308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.699191 23286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:22.699317 23286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "52f2160488f74534b5117515552567f7"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "11451bcd22223b76bffa80c56bc67ca7"
server_key_iv: "c555ac48508e00ff9454fcbeb89e5686"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.699434 23286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:22.728049 23286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:22.731178 23286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:22.731380 23286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:22.732110 23286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:22.732947 23286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:22.732995 23286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.733062 23286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:22.733096 23286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.742151 23286 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:35219
I20260806 16:04:22.742223 23474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:35219 every 8 connection(s)
I20260806 16:04:22.743124 23286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:22.750209 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 23286
I20260806 16:04:22.750362 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:22.750773 13904 external_mini_cluster.cc:1428] Setting key 3b6f31e70808115c95d0aaef41ec568d
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:22.758026 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.744931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:43557 (local address 127.13.148.62:44525)
0806 16:04:22.745073 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:22.745077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:22.745904 (+ 827us) server_negotiation.cc:407] Connection header received
0806 16:04:22.747426 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:22.747430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:22.747494 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:22.747600 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:22.748785 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.749329 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.750548 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.750900 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.751022 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.754043 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:22.754047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:22.754050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:22.754078 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:22.755566 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.756107 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.756110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.756112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:22.756289 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.756638 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.756641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.756642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:22.756861 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:22.756944 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:22.757711 (+ 767us) server_negotiation.cc:299] Negotiation successful
0806 16:04:22.757867 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:04:22.758994 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.745251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.3:43557)
0806 16:04:22.745735 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:22.745780 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:22.747188 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:22.747785 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:22.747792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:22.748160 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:22.748624 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:22.748638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.749459 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.749463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:22.750371 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:22.750384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.751055 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.751058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:22.751262 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.751923 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:22.751944 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:22.753718 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:22.755686 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:22.755691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:22.755703 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:22.755962 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:22.756385 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:22.756388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:22.756389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:22.756526 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:22.756977 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:22.756985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:22.757380 (+ 395us) client_negotiation.cc:769] Sending connection context
0806 16:04:22.757644 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 16:04:22.758048 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260806 16:04:22.760290 23475 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44525
I20260806 16:04:22.760533 23475 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:22.761045 23475 heartbeater.cc:511] Master 127.13.148.62:44525 requested a full tablet report, sending...
I20260806 16:04:22.762104 22774 ts_manager.cc:194] Registered new tserver with Master: 52f2160488f74534b5117515552567f7 (127.13.148.3:35219)
I20260806 16:04:22.762748 22774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:43557
I20260806 16:04:22.764920 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.61.keytab.
Entry for principal oryx/127.13.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.61.keytab.
I20260806 16:04:22.797008 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:34099
--webserver_interface=127.13.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.61.keytab
--principal=oryx/127.13.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:44525,127.13.148.61:34099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:22.914068 23480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:22.914337 23480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:22.914407 23480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:22.918377 23480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:22.918434 23480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:22.918454 23480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:22.918550 23480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:22.918583 23480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:22.923864 23480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:44525,127.13.148.61:34099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.61.keytab
--principal=oryx/127.13.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:34099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
W20260806 16:04:22.924080 23480 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.148.62:44525,127.13.148.61:34099'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 16:04:22.924960 23480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:22.926003 23480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:22.934514 23494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:22.934538 23492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:22.934808 23491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:22.934903 23480 server_base.cc:1034] running on GCE node
I20260806 16:04:22.935496 23480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:22.936653 23480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:22.937845 23480 hybrid_clock.cc:648] HybridClock initialized: now 1786032262937822 us; error 37 us; skew 500 ppm
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:22.940960 23480 init.cc:377] Logged in from keytab as oryx/127.13.148.61@KRBTEST.COM (short username oryx)
I20260806 16:04:22.941865 23480 webserver.cc:460] Webserver started at http://127.13.148.61:40131/ using document root <none> and password file <none>
I20260806 16:04:22.942375 23480 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:22.942436 23480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:22.942744 23480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:22.944737 23480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data/instance:
uuid: "d53ffdbc54aa4654ad2b573a5cb5125f"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "c8931f3a6060284f193be494e2f5e6cb"
server_key_iv: "0d5f7a488247fbd4bf0fd3531ec91d84"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.945240 23480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal/instance:
uuid: "d53ffdbc54aa4654ad2b573a5cb5125f"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "c8931f3a6060284f193be494e2f5e6cb"
server_key_iv: "0d5f7a488247fbd4bf0fd3531ec91d84"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.948894 23480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 16:04:22.951654 23504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:22.952865 23480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:04:22.952993 23480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
uuid: "d53ffdbc54aa4654ad2b573a5cb5125f"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "c8931f3a6060284f193be494e2f5e6cb"
server_key_iv: "0d5f7a488247fbd4bf0fd3531ec91d84"
server_key_version: "encryptionkey@0"
I20260806 16:04:22.953110 23480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:22.968928 23480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:22.972131 23480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:22.972333 23480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:22.979591 23480 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:34099
I20260806 16:04:22.979624 23589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:34099 every 8 connection(s)
I20260806 16:04:22.980723 23480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
I20260806 16:04:22.983804 23591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:22.984263 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 23480
I20260806 16:04:22.988804 23591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
Aug 06 16:04:22 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.61@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:23.000962 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.985836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.13.148.61:34099)
0806 16:04:22.986303 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:22.986318 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:22.986353 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:04:22.987210 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:22.987230 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:22.987650 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:22.988010 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:22.988863 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.989981 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.991197 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.991962 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.992149 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.995420 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:22.995426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:22.995436 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:22.995463 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:22.997428 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.997914 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.997920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.997927 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:22.998110 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.998414 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.998420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.998425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:22.999023 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:22.999271 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:22.999572 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 16:04:22.999913 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":323,"thread_start_us":137,"threads_started":1}
I20260806 16:04:23.001457 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.990661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.13.148.62:44525)
0806 16:04:22.990807 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:22.990813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:22.991124 (+ 311us) server_negotiation.cc:407] Connection header received
0806 16:04:22.991310 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:22.991314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:22.991392 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:22.991466 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:22.992353 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.993389 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.994649 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.994956 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.995087 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.997627 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:22.997630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:22.997632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:22.997657 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:22.999390 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:22.999940 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:22.999944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:22.999946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.000042 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.000380 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.000384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.000386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.000699 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.000812 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.001200 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.001338 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:04:23.002337 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:22.990604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:38628)
0806 16:04:22.990996 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:22.991036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:22.991184 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:22.991546 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:22.991552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:22.992042 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:22.992213 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:22.992221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.993585 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.993588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:22.994460 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:22.994473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:22.995132 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:22.995137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:22.995259 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:22.995847 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:22.995867 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:22.997474 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:22.999542 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:22.999552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:22.999557 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:22.999833 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.000172 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.000175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.000176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.000271 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.000820 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.000826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.000926 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.001120 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.002189 (+ 1069us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":132,"threads_started":1}
I20260806 16:04:23.003870 23591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032262, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.61@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260806 16:04:23.013025 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.004435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.0.0.1:47490)
0806 16:04:23.004598 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.004615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.004707 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.004947 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.004950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.005146 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.005728 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.005738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.006624 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.006628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.007175 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.007183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.007726 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.007730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.007872 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.008424 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.008436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.009820 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.011571 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.011576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.011578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.011834 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.012136 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.012139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.012141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.012234 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.012610 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.012613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.012696 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.012787 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.012891 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":7}
I20260806 16:04:23.013141 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.004485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.13.148.61:34099)
0806 16:04:23.004607 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.004613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.004664 (+ 51us) server_negotiation.cc:407] Connection header received
0806 16:04:23.004797 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.004800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.004844 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.004934 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.005852 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.006401 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.007334 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.007607 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.007730 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.009960 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.009963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.009965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.009988 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.011440 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.011930 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.011933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.011934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.012013 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.012334 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.012336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.012338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.012514 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.012598 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.012874 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.012977 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":47}
I20260806 16:04:23.016309 23591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Bootstrap starting.
I20260806 16:04:23.018788 23591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:23.019595 23591 log.cc:824] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:23.021646 23591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: No bootstrap required, opened a new log
I20260806 16:04:23.025635 23591 raft_consensus.cc:348] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } }
I20260806 16:04:23.025877 23591 raft_consensus.cc:374] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:23.025943 23591 raft_consensus.cc:729] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53ffdbc54aa4654ad2b573a5cb5125f, State: Initialized, Role: FOLLOWER
I20260806 16:04:23.026398 23591 consensus_queue.cc:260] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } }
I20260806 16:04:23.026947 23598 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } } }
I20260806 16:04:23.027064 23598 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:04:23.027755 23591 sys_catalog.cc:564] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: configured and running, proceeding with master startup.
W20260806 16:04:23.032783 23615 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:04:23.032878 23615 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:04:23.035974 23480 master_runner.cc:428] Detected that this master d53ffdbc54aa4654ad2b573a5cb5125f is joining an existing cluster
I20260806 16:04:23.036038 23480 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.148.61:34099
I20260806 16:04:23.046280 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.038147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.13.148.62:44525)
0806 16:04:23.038365 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.038370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.038695 (+ 325us) server_negotiation.cc:407] Connection header received
0806 16:04:23.038836 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.038839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.038884 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.038979 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.039556 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.040105 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.040961 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.041203 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.041317 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.042518 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.042522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.042525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.042550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.044453 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.044989 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.044992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.044994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.045090 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.045412 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.045415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.045417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.045650 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.045746 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.046013 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.046124 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260806 16:04:23.046298 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.038081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:38634)
0806 16:04:23.038600 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.038613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.038740 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.038996 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.038999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.039290 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.039440 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.039446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.040225 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.040232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.040834 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.040843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.041329 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.041332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.041424 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.041974 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.041988 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.042346 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.044572 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.044575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.044578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.044877 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.045197 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.045200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.045203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.045310 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.045744 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.045747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.045845 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.045956 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.046098 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"spinlock_wait_cycles":7424,"thread_start_us":400,"threads_started":1}
I20260806 16:04:23.052186 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.038337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.13.148.61:34099)
0806 16:04:23.038457 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.038498 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.042782 (+ 4284us) server_negotiation.cc:407] Connection header received
0806 16:04:23.042878 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.042882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.042928 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.043016 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.043907 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.044681 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.045667 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.046056 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.046170 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.048202 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.048205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.048208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.048231 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.049954 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.050916 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.050920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.050922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.051020 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.051368 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.051371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.051373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.051582 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.051680 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.051952 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.052061 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:04:23.052559 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.038097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.0.0.1:47498)
0806 16:04:23.042669 (+ 4572us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.042685 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.042794 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.043250 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.043253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.043516 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.043769 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.043778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.044794 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.044799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.045548 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.045558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.046275 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.046279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.046633 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.047581 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.047597 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.048041 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.050560 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.050563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.050565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.050808 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.051153 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.051156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.051158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.051267 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.051724 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.051726 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.051807 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.052326 (+ 519us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.052446 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4500,"mutex_wait_us":14,"thread_start_us":122,"threads_started":1}
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032261, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260806 16:04:23.062773 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.051383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.0.0.1:47512)
0806 16:04:23.051992 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.052051 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.052489 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.053239 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.053245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.053744 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.053887 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.053896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.054918 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.054922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.056017 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.056029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.056970 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.056979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.057097 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.057635 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.057647 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.059177 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.061068 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.061078 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.061084 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.061332 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.061624 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.061627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.061629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.061721 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.062101 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.062110 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.062245 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.062393 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.062605 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":98,"threads_started":1}
I20260806 16:04:23.062783 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.051417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.13.148.61:34099)
0806 16:04:23.051731 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.051740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.052137 (+ 397us) server_negotiation.cc:407] Connection header received
0806 16:04:23.052905 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.052908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.052966 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.053040 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.054058 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.054796 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.056155 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.056584 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.056708 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.059312 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.059315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.059318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.059342 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.060956 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.061438 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.061442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.061444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.061526 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.061815 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.061820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.061822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.061990 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.062099 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.062448 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.062595 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260806 16:04:23.064390 22774 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.148.61:34099: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f"
member_type: NON_VOTER
last_known_addr {
host: "127.13.148.61"
port: 34099
}
attrs {
promote: true
}
}
dest_uuid: "1684ee6e90fd424487d2e316d689569a"
cas_config_opid_index: -1
I20260806 16:04:23.065018 22774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.13.148.60@KRBTEST.COM; defaulting to no policy
I20260806 16:04:23.067447 23631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d53ffdbc54aa4654ad2b573a5cb5125f (127.13.148.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } }
I20260806 16:04:23.068148 23549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:23.069905 23631 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.148.61:34099
I20260806 16:04:23.070317 23631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } }
I20260806 16:04:23.070426 23631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:23.070582 23480 master.cc:537] Master@127.13.148.61:34099 shutting down...
I20260806 16:04:23.070628 23549 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Refusing update from remote peer 1684ee6e90fd424487d2e316d689569a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.60@KRBTEST.COM" created.
I20260806 16:04:23.071916 23631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 16:04:23.074045 22757 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 16:04:23.075344 22757 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a -> Peer d53ffdbc54aa4654ad2b573a5cb5125f (127.13.148.61:34099): Couldn't send request to peer d53ffdbc54aa4654ad2b573a5cb5125f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 16:04:23.076431 23480 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:23.076735 23480 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:23.076849 23480 tablet_replica.cc:330] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: stopping tablet replica
I20260806 16:04:23.082633 23480 master.cc:552] Master@127.13.148.61:34099 shutdown complete.
I20260806 16:04:23.082696 23480 master_runner.cc:305] Clearing existing system tablet
I20260806 16:04:23.082967 23480 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:23.083029 23480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:23.084553 23480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:23.086057 23638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.086323 23480 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:23.086591 23480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:23.086642 23480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
uuid: "d53ffdbc54aa4654ad2b573a5cb5125f"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "c8931f3a6060284f193be494e2f5e6cb"
server_key_iv: "0d5f7a488247fbd4bf0fd3531ec91d84"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.089286 23480 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.60.keytab.
Entry for principal oryx/127.13.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.60.keytab.
I20260806 16:04:23.092746 23480 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 16:04:23.092852 23480 log.cc:1196] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 16:04:23.093036 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:42141
--webserver_interface=127.13.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.60.keytab
--principal=oryx/127.13.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:44525,127.13.148.61:34099,127.13.148.60:42141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:04:23.093139 23480 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Deleting consensus metadata
I20260806 16:04:23.093304 23480 master_runner.cc:315] Copying system tablet from 127.13.148.62:44525
I20260806 16:04:23.094627 23480 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.148.62:44525
I20260806 16:04:23.109339 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.095283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.13.148.62:44525)
0806 16:04:23.095416 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.095421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.100103 (+ 4682us) server_negotiation.cc:407] Connection header received
0806 16:04:23.100196 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.100202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.100272 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.100382 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.100963 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.101893 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.103272 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.103754 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.103897 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.105425 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.105432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.105437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.105472 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.107434 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.107975 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.107979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.107981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.108074 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.108390 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.108393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.108395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.108618 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.108718 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.109028 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.109161 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:04:23.109380 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.095195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:38636)
0806 16:04:23.098672 (+ 3477us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.098689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.098825 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.100421 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.100425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.100675 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.100808 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.100816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.102028 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.102032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.103128 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.103141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.103880 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.103884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.104004 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.104699 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.104716 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.105231 (+ 515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.107563 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.107567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.107570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.107865 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.108177 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.108180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.108183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.108288 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.108725 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.108728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.108841 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.108964 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.109132 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3380,"thread_start_us":86,"threads_started":1}
I20260806 16:04:23.110395 22804 tablet_copy_service.cc:140] P 1684ee6e90fd424487d2e316d689569a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d53ffdbc54aa4654ad2b573a5cb5125f ({username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:38636)
I20260806 16:04:23.110625 22804 tablet_copy_service.cc:161] P 1684ee6e90fd424487d2e316d689569a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d53ffdbc54aa4654ad2b573a5cb5125f at {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:38636: session id = d53ffdbc54aa4654ad2b573a5cb5125f-00000000000000000000000000000000
I20260806 16:04:23.112414 22804 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 16:04:23.114210 23480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 16:04:23.116851 23480 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: tablet copy: Starting download of 0 data blocks...
I20260806 16:04:23.117067 23480 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: tablet copy: Starting download of 1 WAL segments...
I20260806 16:04:23.118902 23480 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 16:04:23.120818 22804 tablet_copy_service.cc:342] P 1684ee6e90fd424487d2e316d689569a: Request end of tablet copy session d53ffdbc54aa4654ad2b573a5cb5125f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:38636
I20260806 16:04:23.120942 22804 tablet_copy_service.cc:433] P 1684ee6e90fd424487d2e316d689569a: ending tablet copy session d53ffdbc54aa4654ad2b573a5cb5125f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d53ffdbc54aa4654ad2b573a5cb5125f
I20260806 16:04:23.128957 23480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:23.130782 23650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:23.131328 23480 server_base.cc:1034] running on GCE node
W20260806 16:04:23.131275 23653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:23.132030 23651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:23.132333 23480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:23.132654 23480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:23.133841 23480 hybrid_clock.cc:648] HybridClock initialized: now 1786032263133826 us; error 30 us; skew 500 ppm
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:23.135828 23480 init.cc:377] Logged in from keytab as oryx/127.13.148.61@KRBTEST.COM (short username oryx)
I20260806 16:04:23.136324 23480 webserver.cc:460] Webserver started at http://127.13.148.61:40771/ using document root <none> and password file <none>
I20260806 16:04:23.136524 23480 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:23.136564 23480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:23.137981 23480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:23.139135 23661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.139380 23480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 16:04:23.139460 23480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
uuid: "d53ffdbc54aa4654ad2b573a5cb5125f"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "c8931f3a6060284f193be494e2f5e6cb"
server_key_iv: "0d5f7a488247fbd4bf0fd3531ec91d84"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.139537 23480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:23.149799 23480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:23.152040 23480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:23.158842 23480 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:34099
I20260806 16:04:23.158972 23743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:34099 every 8 connection(s)
I20260806 16:04:23.159531 23480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
I20260806 16:04:23.161289 23744 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:04:23.162314 23744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Bootstrap starting.
I20260806 16:04:23.172914 23744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:04:23.173270 23744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Bootstrap complete.
I20260806 16:04:23.173825 23744 raft_consensus.cc:348] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } }
I20260806 16:04:23.173965 23744 raft_consensus.cc:729] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d53ffdbc54aa4654ad2b573a5cb5125f, State: Initialized, Role: LEARNER
I20260806 16:04:23.174073 23744 consensus_queue.cc:260] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } }
I20260806 16:04:23.174427 23748 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } }
I20260806 16:04:23.174557 23744 sys_catalog.cc:564] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:23.174712 23748 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: This master's current role is: LEARNER
I20260806 16:04:23.177836 23762 catalog_manager.cc:1259] Loaded cluster ID: 21ed52bc8b6644e9ac2115ac33bbbcf9
I20260806 16:04:23.177934 23762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: loading cluster ID for follower catalog manager: success
I20260806 16:04:23.180236 23762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: acquiring CA information for follower catalog manager: success
I20260806 16:04:23.181622 23762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 16:04:23.225258 23646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:23.225543 23646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:23.225625 23646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:23.229647 23646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:23.229734 23646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:23.229773 23646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:23.229830 23646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:23.229866 23646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:23.235316 23646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:44525,127.13.148.61:34099,127.13.148.60:42141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.60.keytab
--principal=oryx/127.13.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:42141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:23.236406 23646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:23.237444 23646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:23.246543 23769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:23.246582 23768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:23.246707 23646 server_base.cc:1034] running on GCE node
W20260806 16:04:23.246606 23773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:23.247483 23646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:23.248656 23646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:23.249840 23646 hybrid_clock.cc:648] HybridClock initialized: now 1786032263249812 us; error 40 us; skew 500 ppm
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:23.252846 23646 init.cc:377] Logged in from keytab as oryx/127.13.148.60@KRBTEST.COM (short username oryx)
I20260806 16:04:23.253829 23646 webserver.cc:460] Webserver started at http://127.13.148.60:40361/ using document root <none> and password file <none>
I20260806 16:04:23.254433 23646 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:23.254560 23646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:23.254806 23646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:23.257040 23646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data/instance:
uuid: "551afc33da5c4d4296020fbd4716f47f"
format_stamp: "Formatted at 2026-08-06 16:04:23 on dist-test-slave-7kzw"
server_key: "b4719f2bef8aba47ffa8e38dc0311e6b"
server_key_iv: "3a1af581018ba9a0e601ec1b4dfa9c0b"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.257688 23646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal/instance:
uuid: "551afc33da5c4d4296020fbd4716f47f"
format_stamp: "Formatted at 2026-08-06 16:04:23 on dist-test-slave-7kzw"
server_key: "b4719f2bef8aba47ffa8e38dc0311e6b"
server_key_iv: "3a1af581018ba9a0e601ec1b4dfa9c0b"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.261793 23646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:04:23.264482 23783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.265955 23646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 16:04:23.266096 23646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
uuid: "551afc33da5c4d4296020fbd4716f47f"
format_stamp: "Formatted at 2026-08-06 16:04:23 on dist-test-slave-7kzw"
server_key: "b4719f2bef8aba47ffa8e38dc0311e6b"
server_key_iv: "3a1af581018ba9a0e601ec1b4dfa9c0b"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.266218 23646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:23.279979 23646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:23.283120 23646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:23.283326 23646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:23.291112 23646 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:42141
I20260806 16:04:23.291456 23856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:42141 every 8 connection(s)
I20260806 16:04:23.292235 23646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
I20260806 16:04:23.294982 23857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:23.296939 23075 heartbeater.cc:503] Master 127.13.148.62:44525 was elected leader, sending a full tablet report...
I20260806 16:04:23.299368 23857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:04:23.301390 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 23646
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
I20260806 16:04:23.313632 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.301247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.13.148.62:44525)
0806 16:04:23.301397 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.301402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.302245 (+ 843us) server_negotiation.cc:407] Connection header received
0806 16:04:23.303432 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.303435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.303497 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.303642 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.304532 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.305152 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.306314 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.306721 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.306848 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.309784 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.309788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.309790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.309821 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.311385 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.311974 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.311978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.311979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.312078 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.312436 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.312438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.312439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.312612 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.312729 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.313316 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.313438 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:04:23.314527 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.301598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:38644)
0806 16:04:23.302073 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.302124 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.303197 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.303695 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.303700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.304146 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.304379 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.304392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.305286 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.305290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.306188 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.306197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.306883 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.306887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.307077 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.307852 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.307869 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.309620 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.311469 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.311481 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.311493 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.311866 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.312179 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.312181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.312182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.312316 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.312741 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.312748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.313040 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.313241 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.313488 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260806 16:04:23.314519 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.302210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.13.148.60:42141)
0806 16:04:23.302641 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.302653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.302704 (+ 51us) server_negotiation.cc:407] Connection header received
0806 16:04:23.303197 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.303220 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.303451 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.303668 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.304620 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.305652 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.306832 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.307332 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.307440 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.310311 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.310320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.310336 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.310367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.312303 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.312790 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.312795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.312796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.312872 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.313158 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.313161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.313162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.313463 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.313656 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.314016 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.314129 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":174,"threads_started":1}
I20260806 16:04:23.315824 23857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260806 16:04:23.324049 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.316340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.0.0.1:58242)
0806 16:04:23.316489 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.316503 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.316571 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.317154 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.317157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.317283 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.317370 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.317375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.318286 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.318289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.318899 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.318908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.319365 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.319368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.319444 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.319944 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.319956 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.321156 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.322765 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.322767 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.322769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.322959 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.323236 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.323238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.323240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.323324 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.323679 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.323681 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.323762 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.323841 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.323938 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 16:04:23.324113 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.316461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.13.148.61:34099)
0806 16:04:23.316960 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.316963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.316975 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:04:23.317011 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.317013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.317051 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.317125 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.317475 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.318155 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.319019 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.319261 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.319337 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.321285 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.321288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.321290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.321313 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.322647 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.323043 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.323046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.323047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.323125 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.323411 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.323413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.323414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.323595 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.323670 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.323901 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.323997 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":104,"threads_started":1}
I20260806 16:04:23.324903 23857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
I20260806 16:04:23.332345 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.325340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42141 (local address 127.0.0.1:34536)
0806 16:04:23.325484 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.325498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.325580 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.325813 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.325816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.325999 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.326066 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.326071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.326940 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.326947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.327444 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.327451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.327892 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.327896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.327994 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.328372 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.328383 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.329484 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.330965 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.330971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.330973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.331156 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.331431 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.331436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.331437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.331515 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.331854 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.331857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.331943 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.332047 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.332173 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":20}
I20260806 16:04:23.332352 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.325415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.13.148.60:42141)
0806 16:04:23.325517 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.325523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.325537 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:23.325670 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.325672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.325715 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.325803 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.326169 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.326829 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.327555 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.327788 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.327896 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.329611 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.329615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.329617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.329644 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.330864 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.331273 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.331276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.331277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.331343 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.331601 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.331606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.331608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.331763 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.331863 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.332091 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.332234 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":42}
I20260806 16:04:23.335862 23857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Bootstrap starting.
I20260806 16:04:23.338171 23857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:23.338985 23857 log.cc:824] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:23.340969 23857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: No bootstrap required, opened a new log
I20260806 16:04:23.343873 23857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } }
I20260806 16:04:23.344062 23857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:23.344094 23857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551afc33da5c4d4296020fbd4716f47f, State: Initialized, Role: FOLLOWER
I20260806 16:04:23.344449 23857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } }
I20260806 16:04:23.344852 23865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } } }
I20260806 16:04:23.344969 23865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:04:23.345520 23857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: configured and running, proceeding with master startup.
W20260806 16:04:23.350371 23883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:04:23.350484 23883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:04:23.353590 23646 master_runner.cc:428] Detected that this master 551afc33da5c4d4296020fbd4716f47f is joining an existing cluster
I20260806 16:04:23.353669 23646 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.148.60:42141
I20260806 16:04:23.364248 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.356168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.0.0.1:58246)
0806 16:04:23.356951 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.356964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.357086 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.357458 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.357461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.357721 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.357857 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.357865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.358740 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.358744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.359369 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.359377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.359857 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.359860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.359982 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.360519 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.360531 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.360890 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.362722 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.362725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.362727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.362932 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.363219 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.363224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.363233 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.363325 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.363701 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.363704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.363807 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.363927 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.364100 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":33,"thread_start_us":209,"threads_started":1}
I20260806 16:04:23.364312 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.355964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.13.148.61:34099)
0806 16:04:23.356071 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.356076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.357028 (+ 952us) server_negotiation.cc:407] Connection header received
0806 16:04:23.357192 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.357196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.357249 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.357306 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.357990 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.358634 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.359492 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.359732 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.359847 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.361026 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.361030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.361033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.361064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.362606 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.363035 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.363039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.363041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.363126 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.363417 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.363422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.363424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.363603 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.363711 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.364019 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.364154 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:04:23.366586 23865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:23.366755 23865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } }
I20260806 16:04:23.367677 23865 leader_election.cc:288] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1684ee6e90fd424487d2e316d689569a (127.13.148.62:44525), d53ffdbc54aa4654ad2b573a5cb5125f (127.13.148.61:34099)
I20260806 16:04:23.368062 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "551afc33da5c4d4296020fbd4716f47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1684ee6e90fd424487d2e316d689569a" is_pre_election: true
I20260806 16:04:23.368124 23702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "551afc33da5c4d4296020fbd4716f47f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" is_pre_election: true
I20260806 16:04:23.368302 23702 raft_consensus.cc:1799] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 LEARNER]: Handling vote request from an unknown peer 551afc33da5c4d4296020fbd4716f47f
I20260806 16:04:23.368438 23702 raft_consensus.cc:2396] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 551afc33da5c4d4296020fbd4716f47f for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260806 16:04:23.368788 23785 leader_election.cc:302] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 551afc33da5c4d4296020fbd4716f47f; no voters: 1684ee6e90fd424487d2e316d689569a, d53ffdbc54aa4654ad2b573a5cb5125f
I20260806 16:04:23.368883 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 22869
I20260806 16:04:23.371176 23865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:23.372219 23865 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 16:04:23.380555 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.355769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.13.148.62:44525)
0806 16:04:23.355890 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.355896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.356624 (+ 728us) server_negotiation.cc:407] Connection header received
0806 16:04:23.356804 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.356809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.356860 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.356927 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.357537 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.358150 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.359511 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.359877 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.359955 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.376477 (+ 16522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.376485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.376492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.376534 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.378746 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.379201 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.379206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.379210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.379293 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.379561 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.379566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.379570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.379807 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.379875 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.380201 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.380330 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 16:04:23.381129 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.355726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55432)
0806 16:04:23.356534 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.356546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.356683 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.357037 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.357040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.357307 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.357418 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.357424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.358277 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.358282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.359367 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.359378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.360118 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.360121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.360225 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.375595 (+ 15370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.375617 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.376250 (+ 633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.378863 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.378868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.378873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.379106 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.379392 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.379397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.379401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.379479 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.379942 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.379947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.380045 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.380704 (+ 659us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.380874 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":51,"thread_start_us":123,"threads_started":1}
I20260806 16:04:23.394256 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.355735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42141 (local address 127.0.0.1:34546)
0806 16:04:23.356601 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.356616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.356732 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.357037 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.357040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.374811 (+ 17771us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.374968 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.374979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.387029 (+ 12050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.387034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.387925 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.387939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.388597 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.388603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.388724 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.389271 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.389290 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.389805 (+ 515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.392434 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.392439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.392444 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.392719 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.393013 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.393018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.393023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.393107 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.393591 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.393596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.393691 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.393790 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.393942 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":2,"spinlock_wait_cycles":16128,"thread_start_us":125,"threads_started":1}
I20260806 16:04:23.394856 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.356118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.13.148.60:42141)
0806 16:04:23.356235 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.356239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.356697 (+ 458us) server_negotiation.cc:407] Connection header received
0806 16:04:23.356822 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.356825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.356860 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.356926 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.377501 (+ 20575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.386903 (+ 9402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.388047 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.388373 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.388472 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.389959 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.389965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.389972 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.389999 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.392316 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.392831 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.392836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.392841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.392925 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.393194 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.393199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.393204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.393462 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.393557 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.394548 (+ 991us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.394674 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:04:23.397156 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:35813
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:44525
--webserver_port=36777
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099,127.13.148.60:42141 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032261, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
I20260806 16:04:23.403084 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.392917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42141 (local address 127.0.0.1:34562)
0806 16:04:23.393155 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.393169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.393242 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.393954 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.393957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.394156 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.394265 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.394272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.395425 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.395429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.396286 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.396297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.396915 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.396920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.397030 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.397551 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.397566 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.399207 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.401217 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.401221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.401223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.401495 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.401848 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.401852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.401854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.401952 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.402531 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.402534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.402639 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.402758 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.402909 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260806 16:04:23.403752 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.393374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.13.148.60:42141)
0806 16:04:23.393661 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.393669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.393682 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:04:23.393762 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.393765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.393816 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.394376 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.394517 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.395294 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.396422 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.396786 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.396885 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.399342 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.399346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.399348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.399370 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.401098 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.401610 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.401614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.401615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.401742 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.402065 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.402071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.402075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.402350 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.402537 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.403389 (+ 852us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.403563 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":125,"threads_started":1}
I20260806 16:04:23.404750 22774 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.148.60:42141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "551afc33da5c4d4296020fbd4716f47f"
member_type: NON_VOTER
last_known_addr {
host: "127.13.148.60"
port: 42141
}
attrs {
promote: true
}
}
dest_uuid: "1684ee6e90fd424487d2e316d689569a"
cas_config_opid_index: 5
I20260806 16:04:23.405519 22774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } }
I20260806 16:04:23.406816 23629 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 551afc33da5c4d4296020fbd4716f47f (127.13.148.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.407545 23823 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 FOLLOWER]: Refusing update from remote peer 1684ee6e90fd424487d2e316d689569a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 16:04:23.408135 23631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Connected to new peer: Peer: permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:23.408731 23629 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.148.60:42141
I20260806 16:04:23.409018 23629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.409112 23629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:23.409613 23646 master.cc:537] Master@127.13.148.60:42141 shutting down...
I20260806 16:04:23.412770 23823 raft_consensus.cc:858] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } }
W20260806 16:04:23.412881 23823 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } }
I20260806 16:04:23.413295 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.407207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.13.148.61:34099)
0806 16:04:23.407348 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.407353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.407574 (+ 221us) server_negotiation.cc:407] Connection header received
0806 16:04:23.407863 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.407866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.407927 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.407998 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:23.408476 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.409249 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.411641 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.412315 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.412396 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.412544 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.413082 (+ 538us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.413169 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:04:23.413802 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.406955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.0.0.1:58258)
0806 16:04:23.407507 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.407521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.407681 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.408157 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.408159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.408176 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:23.408254 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.408260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.410226 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.410232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.411496 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.411511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.412642 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.412646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.412742 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.412797 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.413596 (+ 799us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.413674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486}
I20260806 16:04:23.414419 23702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 LEARNER]: Refusing update from remote peer 1684ee6e90fd424487d2e316d689569a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 16:04:23.414870 23629 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 16:04:23.414369 23823 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d53ffdbc54aa4654ad2b573a5cb5125f (127.13.148.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } }
I20260806 16:04:23.416667 23702 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 551afc33da5c4d4296020fbd4716f47f (127.13.148.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.417928 23865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1684ee6e90fd424487d2e316d689569a. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } }
I20260806 16:04:23.418022 23865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 16:04:23.418102 23865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } }
I20260806 16:04:23.418156 23865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 16:04:23.418208 23865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } }
I20260806 16:04:23.418274 23865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 16:04:23.418435 23748 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1684ee6e90fd424487d2e316d689569a. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.419497 23748 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: This master's current role is: LEARNER
I20260806 16:04:23.419618 23748 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: true } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.419709 23748 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: This master's current role is: LEARNER
I20260806 16:04:23.421239 23903 mvcc.cc:205] Tried to move back new op lower bound from 7315588145290289152 to 7315588145097240576. Current Snapshot: MvccSnapshot[applied={T|T < 7315588145146511360}]
I20260806 16:04:23.421381 23629 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: attempting to promote NON_VOTER d53ffdbc54aa4654ad2b573a5cb5125f to VOTER
I20260806 16:04:23.421778 23629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } }
I20260806 16:04:23.423336 23702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 LEARNER]: Refusing update from remote peer 1684ee6e90fd424487d2e316d689569a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 16:04:23.423774 23631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 16:04:23.423899 22754 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a -> Peer 551afc33da5c4d4296020fbd4716f47f (127.13.148.60:42141): Couldn't send request to peer 551afc33da5c4d4296020fbd4716f47f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 16:04:23.425457 23646 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260806 16:04:23.425732 23646 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260806 16:04:23.425832 23646 tablet_replica.cc:330] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: stopping tablet replica
I20260806 16:04:23.425863 23631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d53ffdbc54aa4654ad2b573a5cb5125f (127.13.148.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.427703 23631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.427822 23631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:23.427946 23631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.428013 23631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:23.428493 23702 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d53ffdbc54aa4654ad2b573a5cb5125f (127.13.148.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.430238 23909 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.430583 23909 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:04:23.441679 23646 master.cc:552] Master@127.13.148.60:42141 shutdown complete.
I20260806 16:04:23.441736 23646 master_runner.cc:305] Clearing existing system tablet
I20260806 16:04:23.441989 23646 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:23.442046 23646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:23.443470 23646 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:23.444442 23917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.444662 23646 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:23.444867 23646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:23.444924 23646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
uuid: "551afc33da5c4d4296020fbd4716f47f"
format_stamp: "Formatted at 2026-08-06 16:04:23 on dist-test-slave-7kzw"
server_key: "b4719f2bef8aba47ffa8e38dc0311e6b"
server_key_iv: "3a1af581018ba9a0e601ec1b4dfa9c0b"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.446971 23646 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:23.449429 23646 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 16:04:23.449560 23646 log.cc:1196] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 16:04:23.449898 23646 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Deleting consensus metadata
I20260806 16:04:23.450100 23646 master_runner.cc:315] Copying system tablet from 127.13.148.62:44525
I20260806 16:04:23.451309 23646 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.148.62:44525
I20260806 16:04:23.460843 23923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.451900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55434)
0806 16:04:23.452197 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.452212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.452343 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.452619 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.452622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.452847 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.452953 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.452960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.454050 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.454053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.455074 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.455085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.455694 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.455699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.455817 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.456470 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.456491 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.456986 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.459140 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.459144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.459146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.459431 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.459734 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.459737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.459739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.459836 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.460271 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.460274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.460378 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.460488 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.460632 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260806 16:04:23.460860 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.452008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.13.148.62:44525)
0806 16:04:23.452139 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.452143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.452317 (+ 174us) server_negotiation.cc:407] Connection header received
0806 16:04:23.452430 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.452434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.452485 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.452585 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.453083 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.453929 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.455201 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.455543 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.455655 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.457143 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.457147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.457150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.457178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.459025 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.459537 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.459540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.459542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.459635 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.459933 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.459936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.459938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.460161 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.460264 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.460565 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.460683 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:04:23.461652 22804 tablet_copy_service.cc:140] P 1684ee6e90fd424487d2e316d689569a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 551afc33da5c4d4296020fbd4716f47f ({username='oryx', principal='oryx/127.13.148.60@KRBTEST.COM'} at 127.0.0.1:55434)
I20260806 16:04:23.461762 22804 tablet_copy_service.cc:161] P 1684ee6e90fd424487d2e316d689569a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 551afc33da5c4d4296020fbd4716f47f at {username='oryx', principal='oryx/127.13.148.60@KRBTEST.COM'} at 127.0.0.1:55434: session id = 551afc33da5c4d4296020fbd4716f47f-00000000000000000000000000000000
I20260806 16:04:23.462893 22804 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 16:04:23.464251 23646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 16:04:23.466629 23646 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: tablet copy: Starting download of 0 data blocks...
I20260806 16:04:23.466921 23646 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: tablet copy: Starting download of 1 WAL segments...
I20260806 16:04:23.468501 23646 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 16:04:23.470433 22804 tablet_copy_service.cc:342] P 1684ee6e90fd424487d2e316d689569a: Request end of tablet copy session 551afc33da5c4d4296020fbd4716f47f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.148.60@KRBTEST.COM'} at 127.0.0.1:55434
I20260806 16:04:23.470546 22804 tablet_copy_service.cc:433] P 1684ee6e90fd424487d2e316d689569a: ending tablet copy session 551afc33da5c4d4296020fbd4716f47f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 551afc33da5c4d4296020fbd4716f47f
I20260806 16:04:23.477581 23646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:23.479574 23929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:23.479616 23925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:23.482661 23646 server_base.cc:1034] running on GCE node
W20260806 16:04:23.482662 23926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:23.483033 23646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:23.483359 23646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:23.484516 23646 hybrid_clock.cc:648] HybridClock initialized: now 1786032263484492 us; error 42 us; skew 500 ppm
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:23.486212 23646 init.cc:377] Logged in from keytab as oryx/127.13.148.60@KRBTEST.COM (short username oryx)
I20260806 16:04:23.486739 23646 webserver.cc:460] Webserver started at http://127.13.148.60:37855/ using document root <none> and password file <none>
I20260806 16:04:23.486954 23646 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:23.487002 23646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:23.488504 23646 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:23.489555 23937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.489858 23646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:23.489981 23646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
uuid: "551afc33da5c4d4296020fbd4716f47f"
format_stamp: "Formatted at 2026-08-06 16:04:23 on dist-test-slave-7kzw"
server_key: "b4719f2bef8aba47ffa8e38dc0311e6b"
server_key_iv: "3a1af581018ba9a0e601ec1b4dfa9c0b"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.490193 23646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:23.507475 23646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:23.509970 23646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:23.515450 23646 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:42141
I20260806 16:04:23.515481 24024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:42141 every 8 connection(s)
I20260806 16:04:23.516288 23646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
I20260806 16:04:23.518353 24025 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:04:23.519531 24025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Bootstrap starting.
I20260806 16:04:23.526947 23275 heartbeater.cc:503] Master 127.13.148.62:44525 was elected leader, sending a full tablet report...
I20260806 16:04:23.528323 24025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:04:23.528703 24025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: Bootstrap complete.
I20260806 16:04:23.529635 24025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } }
I20260806 16:04:23.529784 24025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 551afc33da5c4d4296020fbd4716f47f, State: Initialized, Role: LEARNER
I20260806 16:04:23.529894 24025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } }
I20260806 16:04:23.530330 24025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:23.533505 24044 catalog_manager.cc:1259] Loaded cluster ID: 21ed52bc8b6644e9ac2115ac33bbbcf9
I20260806 16:04:23.533566 24044 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: loading cluster ID for follower catalog manager: success
I20260806 16:04:23.535162 24028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.535375 24028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: LEARNER
I20260806 16:04:23.535941 24044 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: acquiring CA information for follower catalog manager: success
I20260806 16:04:23.537357 24044 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 16:04:23.544242 23899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:23.544471 23899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:23.544545 23899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:23.548676 23899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:23.548753 23899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:23.548866 23899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:23.554113 23899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:35813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=36777
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099,127.13.148.60:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:23.555183 23899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:23.556216 23899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:23.565932 24049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:23.565917 24054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:23.565917 24050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:23.565984 23899 server_base.cc:1034] running on GCE node
I20260806 16:04:23.566612 23899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:23.567298 23899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:23.568487 23899 hybrid_clock.cc:648] HybridClock initialized: now 1786032263568459 us; error 45 us; skew 500 ppm
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:23.571529 23899 init.cc:377] Logged in from keytab as oryx/127.13.148.1@KRBTEST.COM (short username oryx)
I20260806 16:04:23.572713 23899 webserver.cc:460] Webserver started at http://127.13.148.1:36777/ using document root <none> and password file <none>
I20260806 16:04:23.573285 23899 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:23.573400 23899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:23.577623 23899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:23.579736 24062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.580914 23899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:04:23.581050 23899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "e42cbfbe63404bd39c4850e55be882f5"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "2ea94fa1748a56309b0721cf052ae793"
server_key_iv: "a142f356a9332de96f59ddf99d6ac933"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.581575 23899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:23.605798 23899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:23.608785 23899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:23.609002 23899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:23.609617 23899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:23.610571 23899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:23.610638 23899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.610711 23899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:23.610756 23899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.620486 23899 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:35813
I20260806 16:04:23.620523 24233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:35813 every 8 connection(s)
I20260806 16:04:23.621490 23899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:23.630349 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 23899
I20260806 16:04:23.630592 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 23080
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260806 16:04:23.639204 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.623600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:36015 (local address 127.13.148.62:44525)
0806 16:04:23.623779 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.623784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.624761 (+ 977us) server_negotiation.cc:407] Connection header received
0806 16:04:23.625946 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.625949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.626003 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.626061 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.627497 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.628303 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.629791 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.630136 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.630224 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.633776 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.633780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.633782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.633807 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.636485 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637199 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.637205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.637210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.637303 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637762 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.637765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.637767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.637976 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.638049 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.638886 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.639018 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 16:04:23.639204 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.624005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:58379 (local address 127.13.148.61:34099)
0806 16:04:23.624152 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.624156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.625324 (+ 1168us) server_negotiation.cc:407] Connection header received
0806 16:04:23.625921 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.625925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.625988 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.626090 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.627495 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.628303 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.629790 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.630539 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.630628 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.634859 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.634865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.634873 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.634907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.636787 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637292 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.637296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.637298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.637385 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637740 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.637745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.637746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.637977 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.638049 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.638886 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.639018 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:04:23.639585 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.624084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:51813 (local address 127.13.148.60:42141)
0806 16:04:23.624385 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.624390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.625150 (+ 760us) server_negotiation.cc:407] Connection header received
0806 16:04:23.625944 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.625948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.626003 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.626067 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.627475 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.628175 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.629302 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.629558 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.629672 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.634740 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.634747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.634753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.634791 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.636909 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637598 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.637605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.637610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.637726 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.638145 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.638152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.638156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.638518 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.638616 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.639094 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.639297 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260806 16:04:23.641108 24247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.623954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42141 (local address 127.13.148.1:51813)
0806 16:04:23.625042 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.625055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.625709 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.626265 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.626273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.626813 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.627330 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.627344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.628323 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.628334 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.629170 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.629179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.629718 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.629722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.629889 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.630606 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.630684 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.634539 (+ 3855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.637064 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.637075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.637447 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.637873 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.637884 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.638010 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.638711 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.638717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.638848 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.639749 (+ 901us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.639991 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":356,"thread_start_us":104,"threads_started":1}
I20260806 16:04:23.641103 24246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.623954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.1:36015)
0806 16:04:23.624565 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.624621 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.625709 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.626265 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.626273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.626818 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.627346 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.627354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.628453 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.628456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.629649 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.629659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.630391 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.630395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.630539 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.631211 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.631229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.633591 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.636647 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.636657 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.636679 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.637076 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.637421 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.637430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.637621 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.638156 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.638163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.638578 (+ 415us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.639980 (+ 1402us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.640100 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":212,"spinlock_wait_cycles":7936,"thread_start_us":214,"threads_started":1}
I20260806 16:04:23.641103 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.623953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.13.148.1:58379)
0806 16:04:23.625201 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.625214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.625713 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.626264 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.626271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.626822 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.627344 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.627354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.628453 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.628456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.629649 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.629659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.630998 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.631004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.631121 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.631616 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.631633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.634520 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.636905 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.636909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.636911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.637187 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.637499 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.637501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.637503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.637621 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.638185 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.638187 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.638613 (+ 426us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.639396 (+ 783us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.639898 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":759,"spinlock_wait_cycles":36096,"thread_start_us":106,"threads_started":1}
I20260806 16:04:23.642942 24236 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44525
I20260806 16:04:23.643272 24236 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:23.643778 24236 heartbeater.cc:511] Master 127.13.148.62:44525 requested a full tablet report, sending...
I20260806 16:04:23.642937 24237 heartbeater.cc:347] Connected to a master server at 127.13.148.61:34099
I20260806 16:04:23.644308 24237 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:23.644495 24237 heartbeater.cc:511] Master 127.13.148.61:34099 requested a full tablet report, sending...
I20260806 16:04:23.645074 22774 ts_manager.cc:194] Re-registered known tserver with Master: e42cbfbe63404bd39c4850e55be882f5 (127.13.148.1:35813)
I20260806 16:04:23.645689 22774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:36015
I20260806 16:04:23.647171 23689 ts_manager.cc:194] Registered new tserver with Master: e42cbfbe63404bd39c4850e55be882f5 (127.13.148.1:35813)
I20260806 16:04:23.652184 24235 heartbeater.cc:347] Connected to a master server at 127.13.148.60:42141
I20260806 16:04:23.652309 24235 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:23.652531 24235 heartbeater.cc:511] Master 127.13.148.60:42141 requested a full tablet report, sending...
I20260806 16:04:23.654428 23970 ts_manager.cc:194] Registered new tserver with Master: e42cbfbe63404bd39c4850e55be882f5 (127.13.148.1:35813)
I20260806 16:04:23.658836 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:40533
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:44525
--webserver_port=40217
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099,127.13.148.60:42141 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:04:23.765061 23475 heartbeater.cc:503] Master 127.13.148.62:44525 was elected leader, sending a full tablet report...
W20260806 16:04:23.778733 24251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:23.779003 24251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:23.779047 24251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:23.783155 24251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:23.783216 24251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:23.783279 24251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:23.788589 24251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:40533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=40217
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099,127.13.148.60:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:23.789602 24251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:23.790868 24251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:23.801316 24263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:23.801316 24267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:23.801316 24264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:23.801406 24251 server_base.cc:1034] running on GCE node
I20260806 16:04:23.802135 24251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:23.802924 24251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:23.804117 24251 hybrid_clock.cc:648] HybridClock initialized: now 1786032263804097 us; error 31 us; skew 500 ppm
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:23.807125 24251 init.cc:377] Logged in from keytab as oryx/127.13.148.2@KRBTEST.COM (short username oryx)
I20260806 16:04:23.808086 24251 webserver.cc:460] Webserver started at http://127.13.148.2:40217/ using document root <none> and password file <none>
I20260806 16:04:23.808630 24251 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:23.808704 24251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:23.812925 24251 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 16:04:23.815196 24276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.816567 24251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:23.816694 24251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "1cd0c89391ee4de4948cd407678ac1d0"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "88f07daebc075675d36c61b3735f1b9c"
server_key_iv: "37c65f266079bf1af9185b0bc5e643e6"
server_key_version: "encryptionkey@0"
I20260806 16:04:23.817262 24251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:23.835127 24251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:23.838383 24251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:23.838672 24251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:23.839447 24251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:23.840425 24251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:23.840487 24251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.840530 24251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:23.840546 24251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:23.852257 24251 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:40533
I20260806 16:04:23.852296 24441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:40533 every 8 connection(s)
I20260806 16:04:23.853304 24251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:23.857321 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 24251
I20260806 16:04:23.857548 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 23286
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260806 16:04:23.872900 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.855864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:49523 (local address 127.13.148.60:42141)
0806 16:04:23.856071 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.856075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.857025 (+ 950us) server_negotiation.cc:407] Connection header received
0806 16:04:23.859851 (+ 2826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.859855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.859915 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.860001 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.862025 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.862930 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.864728 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.865070 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.865149 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.868423 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.868431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.868437 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.868471 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.870396 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.871042 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.871047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.871051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.871134 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.871498 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.871503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.871507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.871755 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.871824 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.872494 (+ 670us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.872656 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260806 16:04:23.874962 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.855832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42141 (local address 127.13.148.2:49523)
0806 16:04:23.856940 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.856953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.858433 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.860234 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.860237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.860443 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.860819 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.860840 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.863074 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.863077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.864584 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.864598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.865355 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.865360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.865575 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.866249 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.866274 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.868213 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.870550 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.870558 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.870579 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.870936 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.871232 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.871237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.871241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.871407 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.871899 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.871908 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.872223 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.873433 (+ 1210us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.873810 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":351,"spinlock_wait_cycles":8576,"thread_start_us":138,"threads_started":1}
I20260806 16:04:23.875499 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.855348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:38813 (local address 127.13.148.61:34099)
0806 16:04:23.855483 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.855488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.856809 (+ 1321us) server_negotiation.cc:407] Connection header received
0806 16:04:23.860995 (+ 4186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.861001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.861066 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.861123 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.861850 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.862803 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.865090 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.865628 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.865752 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.869548 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.869556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.869563 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.869610 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.872028 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.873147 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.873155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.873160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.873314 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.873963 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.873969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.873974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.874339 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.874446 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.875031 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.875217 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:04:23.876650 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.855832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.13.148.2:38813)
0806 16:04:23.856648 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.856714 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.860111 (+ 3397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.861275 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.861281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.861503 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.861675 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.861700 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.863052 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.863059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.864876 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.864894 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.865981 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.865989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.866182 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.866930 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.866957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.869313 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.872321 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.872328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.872334 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.872820 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.873653 (+ 833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.873659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.873665 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.873806 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.874579 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.874585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.874754 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.876050 (+ 1296us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.876274 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":83,"thread_start_us":645,"threads_started":1}
I20260806 16:04:23.877017 24445 heartbeater.cc:347] Connected to a master server at 127.13.148.60:42141
I20260806 16:04:23.877267 24445 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:23.877817 24445 heartbeater.cc:511] Master 127.13.148.60:42141 requested a full tablet report, sending...
I20260806 16:04:23.879338 23970 ts_manager.cc:194] Registered new tserver with Master: 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533)
I20260806 16:04:23.880827 24448 heartbeater.cc:347] Connected to a master server at 127.13.148.61:34099
I20260806 16:04:23.880923 24448 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:23.881120 24448 heartbeater.cc:511] Master 127.13.148.61:34099 requested a full tablet report, sending...
I20260806 16:04:23.881824 23689 ts_manager.cc:194] Registered new tserver with Master: 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533)
Aug 06 16:04:23 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032263, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:23.888607 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:35219
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:44525
--webserver_port=43333
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099,127.13.148.60:42141 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:04:23.889794 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.855401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33893 (local address 127.13.148.62:44525)
0806 16:04:23.855518 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.855523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.857272 (+ 1749us) server_negotiation.cc:407] Connection header received
0806 16:04:23.858746 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.858752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.858825 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.858894 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:23.862553 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.863301 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.880876 (+ 17575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.881365 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.881513 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.885298 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:23.885305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:23.885312 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:23.885352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:23.887660 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.888276 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.888279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.888281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.888367 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:23.888808 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:23.888814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:23.888818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:23.889036 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:23.889122 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.889529 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.889676 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:04:23.889974 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.855832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.2:33893)
0806 16:04:23.857146 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.857158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.858195 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.859354 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.859368 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.859978 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:23.861858 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.861869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.880075 (+ 18206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.880079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.880689 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.880701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.881714 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.881721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.881870 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.883261 (+ 1391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:23.883283 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:23.885113 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:23.887829 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.887836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.887838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.888165 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.888470 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:23.888477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:23.888482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:23.888596 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:23.889145 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:23.889152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:23.889282 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.889408 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.889591 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"mutex_wait_us":1559,"spinlock_wait_cycles":15360,"thread_start_us":121,"threads_started":1}
I20260806 16:04:23.890833 24446 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44525
I20260806 16:04:23.890924 24446 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:23.891204 24446 heartbeater.cc:511] Master 127.13.148.62:44525 requested a full tablet report, sending...
I20260806 16:04:23.892030 22774 ts_manager.cc:194] Re-registered known tserver with Master: 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533)
I20260806 16:04:23.892906 22774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:33893
I20260806 16:04:23.986271 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.982437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42141 (local address 127.0.0.1:34578)
0806 16:04:23.982761 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:23.982786 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:23.982949 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:23.983277 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:23.983281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:23.983303 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:23.983423 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.983429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.984217 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.984221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.985109 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:23.985128 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.985840 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.985843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:23.985934 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.985958 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:04:23.986025 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 16:04:23.986111 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":85,"threads_started":1}
I20260806 16:04:23.986266 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:23.982562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.13.148.60:42141)
0806 16:04:23.982732 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:23.982737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:23.982881 (+ 144us) server_negotiation.cc:407] Connection header received
0806 16:04:23.983082 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:23.983085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:23.983149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:23.983257 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:23.983538 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.984092 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.985244 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:23.985687 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:23.985830 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:23.985993 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:23.986035 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 16:04:23.986118 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 16:04:23.987036 23987 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 16:04:23.987457 24028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1684ee6e90fd424487d2e316d689569a. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:23.987669 24028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: LEARNER
W20260806 16:04:24.008852 24453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:24.009132 24453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:24.009176 24453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:24.013159 24453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:24.013214 24453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:24.013279 24453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:24.018896 24453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:35219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=43333
--tserver_master_addrs=127.13.148.62:44525,127.13.148.61:34099,127.13.148.60:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:24.019936 24453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:24.020990 24453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:24.031070 24462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:24.031198 24463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:24.031320 24465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:24.031519 24453 server_base.cc:1034] running on GCE node
I20260806 16:04:24.031996 24453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:24.032775 24453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:24.034047 24453 hybrid_clock.cc:648] HybridClock initialized: now 1786032264034031 us; error 31 us; skew 500 ppm
Aug 06 16:04:24 dist-test-slave-7kzw krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032264, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:24.037357 24453 init.cc:377] Logged in from keytab as oryx/127.13.148.3@KRBTEST.COM (short username oryx)
I20260806 16:04:24.038331 24453 webserver.cc:460] Webserver started at http://127.13.148.3:43333/ using document root <none> and password file <none>
I20260806 16:04:24.038980 24453 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:24.039098 24453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:24.043167 24453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:04:24.045260 24477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:24.046571 24453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:04:24.046756 24453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "52f2160488f74534b5117515552567f7"
format_stamp: "Formatted at 2026-08-06 16:04:22 on dist-test-slave-7kzw"
server_key: "11451bcd22223b76bffa80c56bc67ca7"
server_key_iv: "c555ac48508e00ff9454fcbeb89e5686"
server_key_version: "encryptionkey@0"
I20260806 16:04:24.047345 24453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:24.063519 24453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:24.066681 24453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:24.066893 24453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:24.067497 24453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:24.068375 24453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:24.068461 24453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:24.068568 24453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:24.068617 24453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:24.078352 24453 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:35219
I20260806 16:04:24.078388 24651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:35219 every 8 connection(s)
I20260806 16:04:24.079370 24453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:24.087425 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 24453
Aug 06 16:04:24 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032264, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
Aug 06 16:04:24 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032264, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
Aug 06 16:04:24 dist-test-slave-7kzw krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032264, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260806 16:04:24.098707 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.081247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:47611 (local address 127.13.148.60:42141)
0806 16:04:24.081388 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.081393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.082808 (+ 1415us) server_negotiation.cc:407] Connection header received
0806 16:04:24.083821 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.083825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.083882 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.083935 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.085426 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.086277 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.087881 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.088216 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.088316 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.093581 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.093587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.093590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.093617 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.095509 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.096227 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.096231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.096233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.096331 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.097167 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.097171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.097175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.097432 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.097536 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.098344 (+ 808us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.098505 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:04:24.099103 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.081352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58065 (local address 127.13.148.62:44525)
0806 16:04:24.081501 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.081506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.082551 (+ 1045us) server_negotiation.cc:407] Connection header received
0806 16:04:24.083804 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.083807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.083869 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.083950 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.085433 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.086284 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.087773 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.088111 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.088194 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.093073 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.093076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.093080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.093112 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.095907 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.097273 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.097277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.097279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.097368 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.097659 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.097662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.097664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.097882 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.097968 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.098817 (+ 849us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.098943 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:04:24.100081 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.082015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:59011 (local address 127.13.148.61:34099)
0806 16:04:24.082140 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.082145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.082890 (+ 745us) server_negotiation.cc:407] Connection header received
0806 16:04:24.083821 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.083825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.083880 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.083935 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.085410 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.085989 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.087764 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.088123 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.088203 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.093623 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.093628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.093631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.093661 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.095608 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.098542 (+ 2934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.098546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.098548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.098634 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.099114 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.099118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.099120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.099356 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.099448 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.099824 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.099943 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:04:24.100103 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.081698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42141 (local address 127.13.148.3:47611)
0806 16:04:24.082705 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:24.082718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:24.083582 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:24.084129 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:24.084138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:24.084699 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:24.085255 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.085263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.086393 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.086396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.087729 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.087741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.088517 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.088523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.088733 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.089532 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:24.089556 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:24.093403 (+ 3847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:24.095663 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:24.095671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:24.095693 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:24.096106 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:24.096444 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:24.096447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:24.096449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:24.097027 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:24.097673 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:24.097683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:24.098040 (+ 357us) client_negotiation.cc:769] Sending connection context
0806 16:04:24.098859 (+ 819us) client_negotiation.cc:240] Negotiation successful
0806 16:04:24.099014 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":283,"spinlock_wait_cycles":829568,"thread_start_us":147,"threads_started":1}
I20260806 16:04:24.100351 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.081685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.3:58065)
0806 16:04:24.082381 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:24.082436 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:24.083598 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:24.084130 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:24.084137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:24.084666 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:24.085228 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.085243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.086819 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.086823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.087639 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.087657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.088401 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.088404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.088735 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.089871 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:24.089886 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:24.092888 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:24.096614 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:24.096620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:24.096625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:24.097151 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:24.097467 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:24.097470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:24.097471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:24.097557 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:24.098019 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:24.098022 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:24.098137 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 16:04:24.098269 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 16:04:24.098921 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":78,"spinlock_wait_cycles":4352,"thread_start_us":154,"threads_started":1}
I20260806 16:04:24.101378 24652 heartbeater.cc:347] Connected to a master server at 127.13.148.60:42141
I20260806 16:04:24.101378 24654 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44525
I20260806 16:04:24.101667 24652 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:24.101666 24654 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:24.102358 24652 heartbeater.cc:511] Master 127.13.148.60:42141 requested a full tablet report, sending...
I20260806 16:04:24.102360 24654 heartbeater.cc:511] Master 127.13.148.62:44525 requested a full tablet report, sending...
I20260806 16:04:24.103576 22774 ts_manager.cc:194] Re-registered known tserver with Master: 52f2160488f74534b5117515552567f7 (127.13.148.3:35219)
I20260806 16:04:24.103631 23970 ts_manager.cc:194] Registered new tserver with Master: 52f2160488f74534b5117515552567f7 (127.13.148.3:35219)
I20260806 16:04:24.103964 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.081911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.13.148.3:59011)
0806 16:04:24.082784 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:24.082797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:24.083582 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:24.084129 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:24.084138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:24.084691 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:24.085255 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.085263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.086187 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.086190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.087614 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.087625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.088401 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.088404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.088753 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.090033 (+ 1280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:24.090047 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:24.093459 (+ 3412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:24.095745 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:24.095751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:24.095755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:24.098362 (+ 2607us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x360006b8. stack: 00007fa00bdcfc78 00007fa00bdd01a9 00007fa00b9921d7 00000000005984a6 00000000005984f9 000000000059c488 00007fa00bdcc2ab 00007fa00bdcbad0 00007fa0104a40ee 00007fa010427cbe 00007fa0104272c8 00007fa010426146 00007fa010422e17 00007fa010463832 00007fa01046468c 00007fa01047a0d3
0806 16:04:24.098381 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:24.098906 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:24.098909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:24.098911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:24.099001 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:24.099515 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:24.099518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:24.099615 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 16:04:24.100317 (+ 702us) client_negotiation.cc:240] Negotiation successful
0806 16:04:24.100473 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":269,"spinlock_wait_cycles":2667904,"thread_start_us":113,"threads_started":1}
I20260806 16:04:24.104610 22774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:58065
I20260806 16:04:24.104668 24655 heartbeater.cc:347] Connected to a master server at 127.13.148.61:34099
I20260806 16:04:24.104748 24655 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:24.104952 24655 heartbeater.cc:511] Master 127.13.148.61:34099 requested a full tablet report, sending...
I20260806 16:04:24.105556 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.089800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.13.148.61:34099)
0806 16:04:24.092487 (+ 2687us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.092492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.092505 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:04:24.092542 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.092545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.092590 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.092685 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.093630 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.094488 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.095758 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.096119 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.096219 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.098137 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.098141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.098143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.098167 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.101746 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.102274 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.102277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.102279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.102370 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.103000 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.103003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.103005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.103242 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.104607 (+ 1365us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.105275 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.105400 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2609,"thread_start_us":71,"threads_started":1}
I20260806 16:04:24.105944 23689 ts_manager.cc:194] Registered new tserver with Master: 52f2160488f74534b5117515552567f7 (127.13.148.3:35219)
I20260806 16:04:24.108266 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.089952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.13.148.60:42141)
0806 16:04:24.091510 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.091515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.091531 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:04:24.091569 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.091572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.091619 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.091715 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.092607 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.096507 (+ 3900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.101411 (+ 4904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.101896 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.101981 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.103656 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.103660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.103666 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.103697 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.106338 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.106860 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.106863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.106865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.106955 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.107283 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.107286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.107288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.107470 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.107620 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.107859 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.108105 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1471,"thread_start_us":90,"threads_started":1}
I20260806 16:04:24.111213 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.089497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.13.148.62:44525)
0806 16:04:24.091993 (+ 2496us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.091998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.092013 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:04:24.092068 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.092071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.092128 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.092222 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.094025 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.096503 (+ 2478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.098566 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.106567 (+ 8001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.106684 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.107976 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.107979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.107982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.108006 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.109539 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.109943 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.109945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.109946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.110025 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.110326 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.110328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.110329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.110573 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.110680 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.110862 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.110966 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":2417,"thread_start_us":104,"threads_started":1}
I20260806 16:04:24.114042 22774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:24.117080 22774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:24.119679 23900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: true } } }
I20260806 16:04:24.119827 23900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:24.121694 23900 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: attempting to promote NON_VOTER 551afc33da5c4d4296020fbd4716f47f to VOTER
I20260806 16:04:24.122071 23900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } }
I20260806 16:04:24.123340 23987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 LEARNER]: Refusing update from remote peer 1684ee6e90fd424487d2e316d689569a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 16:04:24.123422 23702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Refusing update from remote peer 1684ee6e90fd424487d2e316d689569a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 16:04:24.123932 24677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Connected to new peer: Peer: permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 16:04:24.124101 23900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 16:04:24.126003 24676 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 551afc33da5c4d4296020fbd4716f47f (127.13.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:24.126399 23702 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 551afc33da5c4d4296020fbd4716f47f (127.13.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:24.127653 24676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:24.127771 24676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:24.127918 24676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:24.127997 24676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:24.128124 24676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:24.128199 24676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:24.128334 24688 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:24.128494 24688 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:04:24.130456 23987 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 551afc33da5c4d4296020fbd4716f47f (127.13.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:24.132077 24028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1684ee6e90fd424487d2e316d689569a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:24.132261 24028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:04:24.141716 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.135492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:35813 (local address 127.0.0.1:60498)
0806 16:04:24.135755 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:24.135773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:24.135857 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:24.136700 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:24.136703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:24.136723 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:24.136834 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.136841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.138768 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.138772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.140085 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.140096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.141229 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.141233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.141385 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.141410 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:04:24.141482 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 16:04:24.141566 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1}
I20260806 16:04:24.141782 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.135793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.13.148.1:35813)
0806 16:04:24.136136 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.136152 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.136172 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:04:24.136278 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.136286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.136526 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.136728 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:24.137244 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.138621 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.140273 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.141093 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.141231 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.141444 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.141523 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.141625 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":126,"threads_started":1}
I20260806 16:04:24.142378 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.135867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:40533 (local address 127.0.0.1:49412)
0806 16:04:24.136985 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:24.137000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:24.137120 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:24.137879 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:24.137882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:24.137901 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:24.137994 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.138003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.139235 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.139239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.140612 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.140622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.141934 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.141938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.142047 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.142072 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:04:24.142146 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 16:04:24.142212 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":119,"threads_started":1}
I20260806 16:04:24.142846 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.136338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.13.148.2:40533)
0806 16:04:24.136708 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.136722 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.137101 (+ 379us) server_negotiation.cc:407] Connection header received
0806 16:04:24.137405 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.137411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.137743 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.137904 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:24.138104 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.139135 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.140793 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.141591 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.141706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.141868 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.142596 (+ 728us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.142693 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260806 16:04:24.143035 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.136755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:35219 (local address 127.0.0.1:33268)
0806 16:04:24.137702 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:24.137716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:24.137805 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:24.138576 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:24.138579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:24.138599 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:24.138688 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.138694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.139920 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.139926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.140975 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.140986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.142574 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.142578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.142664 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.142687 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:04:24.142752 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 16:04:24.142814 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":95,"threads_started":1}
I20260806 16:04:24.143568 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.137201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.13.148.3:35219)
0806 16:04:24.138080 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.138093 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.138115 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:04:24.138235 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.138240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.138413 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.138563 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:24.138822 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.139784 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.141125 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.142299 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.142406 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.143292 (+ 886us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.143341 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.143419 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":105,"threads_started":1}
I20260806 16:04:24.145246 24137 tablet_service.cc:1467] Processing CreateTablet for tablet b58c40338f74480b9c86be170e0968a0 (DEFAULT_TABLE table=test-table [id=ba6330f4612549318a43c380e10fe20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:24.145654 24346 tablet_service.cc:1467] Processing CreateTablet for tablet b58c40338f74480b9c86be170e0968a0 (DEFAULT_TABLE table=test-table [id=ba6330f4612549318a43c380e10fe20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:24.146489 24137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58c40338f74480b9c86be170e0968a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:24.146728 24346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58c40338f74480b9c86be170e0968a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:24.150274 24547 tablet_service.cc:1467] Processing CreateTablet for tablet b58c40338f74480b9c86be170e0968a0 (DEFAULT_TABLE table=test-table [id=ba6330f4612549318a43c380e10fe20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:24.151490 24547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58c40338f74480b9c86be170e0968a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:24.155512 24696 tablet_bootstrap.cc:492] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: Bootstrap starting.
I20260806 16:04:24.156510 24699 tablet_bootstrap.cc:492] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: Bootstrap starting.
I20260806 16:04:24.158093 24696 tablet_bootstrap.cc:654] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:24.158579 24699 tablet_bootstrap.cc:654] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:24.159411 24699 log.cc:824] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:24.159883 24700 tablet_bootstrap.cc:492] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Bootstrap starting.
I20260806 16:04:24.160502 24696 log.cc:824] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:24.161993 24700 tablet_bootstrap.cc:654] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:24.162562 24699 tablet_bootstrap.cc:492] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: No bootstrap required, opened a new log
I20260806 16:04:24.162693 24696 tablet_bootstrap.cc:492] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: No bootstrap required, opened a new log
I20260806 16:04:24.162734 24699 ts_tablet_manager.cc:1397] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260806 16:04:24.162889 24696 ts_tablet_manager.cc:1397] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 16:04:24.162874 24700 log.cc:824] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:24.164937 24700 tablet_bootstrap.cc:492] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: No bootstrap required, opened a new log
I20260806 16:04:24.165119 24700 ts_tablet_manager.cc:1397] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:04:24.166404 24699 raft_consensus.cc:348] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.166689 24699 raft_consensus.cc:374] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:24.166741 24699 raft_consensus.cc:729] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42cbfbe63404bd39c4850e55be882f5, State: Initialized, Role: FOLLOWER
I20260806 16:04:24.167176 24699 consensus_queue.cc:260] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.168109 24699 ts_tablet_manager.cc:1428] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:04:24.168437 24236 heartbeater.cc:503] Master 127.13.148.62:44525 was elected leader, sending a full tablet report...
I20260806 16:04:24.171821 24696 raft_consensus.cc:348] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.172109 24696 raft_consensus.cc:374] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:24.172178 24696 raft_consensus.cc:729] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd0c89391ee4de4948cd407678ac1d0, State: Initialized, Role: FOLLOWER
I20260806 16:04:24.172676 24696 consensus_queue.cc:260] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.173839 24696 ts_tablet_manager.cc:1428] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: Time spent starting tablet: real 0.011s user 0.001s sys 0.006s
I20260806 16:04:24.174561 24446 heartbeater.cc:503] Master 127.13.148.62:44525 was elected leader, sending a full tablet report...
I20260806 16:04:24.176406 24700 raft_consensus.cc:348] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.176703 24700 raft_consensus.cc:374] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:24.176774 24700 raft_consensus.cc:729] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f2160488f74534b5117515552567f7, State: Initialized, Role: FOLLOWER
I20260806 16:04:24.177234 24700 consensus_queue.cc:260] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.178023 24654 heartbeater.cc:503] Master 127.13.148.62:44525 was elected leader, sending a full tablet report...
I20260806 16:04:24.178375 24700 ts_tablet_manager.cc:1428] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260806 16:04:24.289234 24706 raft_consensus.cc:482] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:24.289470 24706 raft_consensus.cc:504] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.290825 24706 leader_election.cc:288] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533), 52f2160488f74534b5117515552567f7 (127.13.148.3:35219)
I20260806 16:04:24.294986 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.291104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:35219 (local address 127.13.148.1:41159)
0806 16:04:24.291624 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:24.291636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:24.291758 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:24.292045 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:24.292049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:24.292070 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:24.292181 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.292185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.292970 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.292973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.293865 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.293873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.294533 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.294537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.294637 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.294666 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 16:04:24.294740 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 16:04:24.294791 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":85,"thread_start_us":126,"threads_started":1}
I20260806 16:04:24.294998 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.291239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:41159 (local address 127.13.148.3:35219)
0806 16:04:24.291380 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.291386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.291723 (+ 337us) server_negotiation.cc:407] Connection header received
0806 16:04:24.291866 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.291869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.291951 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.292053 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:24.292296 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.292826 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.294002 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.294397 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.294552 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.294611 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.294782 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.294871 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:04:24.295118 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.291337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:55203 (local address 127.13.148.2:40533)
0806 16:04:24.291469 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.291473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.291610 (+ 137us) server_negotiation.cc:407] Connection header received
0806 16:04:24.291810 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.291815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.291873 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.291972 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:24.292326 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.292822 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.294047 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.294421 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.294571 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.294628 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.294903 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.294981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:04:24.295135 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.291104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:40533 (local address 127.13.148.1:55203)
0806 16:04:24.291499 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:24.291512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:24.291652 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:24.291990 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:24.291996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:24.292019 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:24.292178 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.292184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.292950 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.292955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.293884 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:24.293894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.294602 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.294606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:24.294744 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.294779 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 16:04:24.294863 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 16:04:24.294925 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":64,"thread_start_us":131,"threads_started":1}
I20260806 16:04:24.295948 24371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58c40338f74480b9c86be170e0968a0" candidate_uuid: "e42cbfbe63404bd39c4850e55be882f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd0c89391ee4de4948cd407678ac1d0" is_pre_election: true
I20260806 16:04:24.295995 24579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58c40338f74480b9c86be170e0968a0" candidate_uuid: "e42cbfbe63404bd39c4850e55be882f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f2160488f74534b5117515552567f7" is_pre_election: true
I20260806 16:04:24.296277 24371 raft_consensus.cc:2454] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e42cbfbe63404bd39c4850e55be882f5 in term 0.
I20260806 16:04:24.296277 24579 raft_consensus.cc:2454] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e42cbfbe63404bd39c4850e55be882f5 in term 0.
I20260806 16:04:24.296767 24072 leader_election.cc:302] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cd0c89391ee4de4948cd407678ac1d0, 52f2160488f74534b5117515552567f7, e42cbfbe63404bd39c4850e55be882f5; no voters:
I20260806 16:04:24.297093 24706 raft_consensus.cc:2781] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:24.297199 24706 raft_consensus.cc:482] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:24.297284 24706 raft_consensus.cc:3037] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:24.298382 24706 raft_consensus.cc:504] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.298760 24706 leader_election.cc:288] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [CANDIDATE]: Term 1 election: Requested vote from peers 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533), 52f2160488f74534b5117515552567f7 (127.13.148.3:35219)
I20260806 16:04:24.299149 24371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58c40338f74480b9c86be170e0968a0" candidate_uuid: "e42cbfbe63404bd39c4850e55be882f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd0c89391ee4de4948cd407678ac1d0"
I20260806 16:04:24.299263 24579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58c40338f74480b9c86be170e0968a0" candidate_uuid: "e42cbfbe63404bd39c4850e55be882f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f2160488f74534b5117515552567f7"
I20260806 16:04:24.299289 24371 raft_consensus.cc:3037] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:24.299391 24579 raft_consensus.cc:3037] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:24.300639 24579 raft_consensus.cc:2454] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e42cbfbe63404bd39c4850e55be882f5 in term 1.
I20260806 16:04:24.300639 24371 raft_consensus.cc:2454] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e42cbfbe63404bd39c4850e55be882f5 in term 1.
I20260806 16:04:24.300964 24067 leader_election.cc:302] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cd0c89391ee4de4948cd407678ac1d0, 52f2160488f74534b5117515552567f7, e42cbfbe63404bd39c4850e55be882f5; no voters:
I20260806 16:04:24.301200 24706 raft_consensus.cc:2781] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:24.301398 24706 raft_consensus.cc:686] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 1 LEADER]: Becoming Leader. State: Replica: e42cbfbe63404bd39c4850e55be882f5, State: Running, Role: LEADER
I20260806 16:04:24.301728 24706 consensus_queue.cc:237] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } }
I20260806 16:04:24.305137 22774 catalog_manager.cc:5495] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e42cbfbe63404bd39c4850e55be882f5 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "e42cbfbe63404bd39c4850e55be882f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } health_report { overall_health: UNKNOWN } } }
W20260806 16:04:24.330772 24656 tablet.cc:2369] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:04:24.332950 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.329275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.13.148.1:35813)
0806 16:04:24.329405 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.329408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.329421 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:04:24.329496 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.329498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.329551 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.329642 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:24.329890 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.330448 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.331414 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.331706 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.331790 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.332079 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:24.332542 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:24.332635 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.332723 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.332775 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:04:24.338842 24579 raft_consensus.cc:1261] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 1 FOLLOWER]: Refusing update from remote peer e42cbfbe63404bd39c4850e55be882f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:24.338845 24371 raft_consensus.cc:1261] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 1 FOLLOWER]: Refusing update from remote peer e42cbfbe63404bd39c4850e55be882f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:24.339622 24714 consensus_queue.cc:1035] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:24.339833 24706 consensus_queue.cc:1035] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:24.352080 24721 mvcc.cc:205] Tried to move back new op lower bound from 7315588154724646912 to 7315588154582536192. Current Snapshot: MvccSnapshot[applied={T|T < 7315588154724646912 or (T in {7315588154724646912})}]
I20260806 16:04:24.352330 24720 mvcc.cc:205] Tried to move back new op lower bound from 7315588154724646912 to 7315588154582536192. Current Snapshot: MvccSnapshot[applied={T|T < 7315588154724646912}]
I20260806 16:04:24.360319 22773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:24.360471 22773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:24.365556 22773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1684ee6e90fd424487d2e316d689569a: Sending DeleteTablet for 3 replicas of tablet b58c40338f74480b9c86be170e0968a0
I20260806 16:04:24.366313 24137 tablet_service.cc:1514] Processing DeleteTablet for tablet b58c40338f74480b9c86be170e0968a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:24 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:60498
I20260806 16:04:24.366484 24346 tablet_service.cc:1514] Processing DeleteTablet for tablet b58c40338f74480b9c86be170e0968a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:24 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:49412
I20260806 16:04:24.366873 24738 tablet_replica.cc:330] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: stopping tablet replica
I20260806 16:04:24.367058 24547 tablet_service.cc:1514] Processing DeleteTablet for tablet b58c40338f74480b9c86be170e0968a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:24 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:33268
I20260806 16:04:24.367242 24738 raft_consensus.cc:2229] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:24.367605 24740 tablet_replica.cc:330] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: stopping tablet replica
I20260806 16:04:24.367697 24738 raft_consensus.cc:2258] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:24.368057 24740 raft_consensus.cc:2229] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:24.368449 24740 raft_consensus.cc:2258] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:24.369252 24738 ts_tablet_manager.cc:1905] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:24.370142 24740 ts_tablet_manager.cc:1905] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:24.372189 24738 ts_tablet_manager.cc:1918] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:24.372300 24738 log.cc:1196] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/b58c40338f74480b9c86be170e0968a0
I20260806 16:04:24.372449 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 22729
I20260806 16:04:24.372632 24738 ts_tablet_manager.cc:1939] T b58c40338f74480b9c86be170e0968a0 P e42cbfbe63404bd39c4850e55be882f5: Deleting consensus metadata
I20260806 16:04:24.381767 24740 ts_tablet_manager.cc:1918] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:24.382015 24740 log.cc:1196] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/b58c40338f74480b9c86be170e0968a0
I20260806 16:04:24.382491 24740 ts_tablet_manager.cc:1939] T b58c40338f74480b9c86be170e0968a0 P 52f2160488f74534b5117515552567f7: Deleting consensus metadata
I20260806 16:04:24.383208 24739 tablet_replica.cc:330] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: stopping tablet replica
I20260806 16:04:24.383581 24739 raft_consensus.cc:2229] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:24.383941 24739 raft_consensus.cc:2258] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:24.385598 24739 ts_tablet_manager.cc:1905] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:24.388530 24739 ts_tablet_manager.cc:1918] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:24.388736 24739 log.cc:1196] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/b58c40338f74480b9c86be170e0968a0
W20260806 16:04:24.388854 24068 connection.cc:535] server connection from 127.0.0.1:60498 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 16:04:24.389063 24739 ts_tablet_manager.cc:1939] T b58c40338f74480b9c86be170e0968a0 P 1cd0c89391ee4de4948cd407678ac1d0: Deleting consensus metadata
W20260806 16:04:24.390173 24277 connection.cc:441] server connection from 127.0.0.1:49412 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49412 (request call id 1) could send its response
W20260806 16:04:24.393035 24484 connection.cc:441] server connection from 127.0.0.1:33268 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33268 (request call id 1) could send its response
I20260806 16:04:24.395311 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.370972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.13.148.61:34099)
0806 16:04:24.371094 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.371101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.382888 (+ 11787us) server_negotiation.cc:407] Connection header received
0806 16:04:24.383034 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.383040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.383114 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.383212 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.384628 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.385570 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.387114 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.387583 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.387700 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.390198 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.390201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.390204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.390238 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.392405 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.392900 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.392904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.392905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.393001 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.393323 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.393327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.393329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.393545 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.393680 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.395028 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.395149 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 16:04:24.404481 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.393946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.13.148.61:34099)
0806 16:04:24.394057 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.394062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.394575 (+ 513us) server_negotiation.cc:407] Connection header received
0806 16:04:24.394742 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.394745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.394792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.394847 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.396047 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.396766 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.397854 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.398192 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.398258 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.400327 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.400331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.400333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.400359 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.402403 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.403035 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.403039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.403040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.403122 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.403441 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.403447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.403452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.403681 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.403799 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.404181 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.404301 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:04:24.404817 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.393182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.13.148.60:42141)
0806 16:04:24.393361 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.393366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.394123 (+ 757us) server_negotiation.cc:407] Connection header received
0806 16:04:24.394314 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.394317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.394377 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.394434 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.395781 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.396587 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.397854 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.398182 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.398258 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.400328 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.400331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.400333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.400358 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.402921 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.403398 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.403404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.403407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.403495 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.403807 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.403813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.403817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.404031 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.404136 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.404525 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.404642 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 16:04:24.405169 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:24.369707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.13.148.60:42141)
0806 16:04:24.371080 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:24.371084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:24.394396 (+ 23312us) server_negotiation.cc:407] Connection header received
0806 16:04:24.394615 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:24.394618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:24.394676 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:24.394725 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:24.396755 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.397528 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.399316 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:24.399691 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:24.399763 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:24.401328 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:24.401332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:24.401335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:24.401360 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:24.403193 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.403674 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.403677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.403679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.403763 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:24.404050 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:24.404054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:24.404055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:24.404247 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:24.404343 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:24.404693 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 16:04:24.405017 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129}
I20260806 16:04:25.353863 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.353384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.3:34531)
0806 16:04:25.353655 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.353759 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260806 16:04:25.354141 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.353705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.1:56043)
0806 16:04:25.353931 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.354038 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
W20260806 16:04:25.354364 24654 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:44525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
W20260806 16:04:25.354748 24236 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:44525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
I20260806 16:04:25.354959 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.354373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.2:52801)
0806 16:04:25.354749 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.354861 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":120,"threads_started":1}
W20260806 16:04:25.355232 24446 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:44525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
I20260806 16:04:25.864789 24756 raft_consensus.cc:482] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1684ee6e90fd424487d2e316d689569a)
I20260806 16:04:25.864979 24756 raft_consensus.cc:504] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } }
I20260806 16:04:25.865962 24757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1684ee6e90fd424487d2e316d689569a)
I20260806 16:04:25.866011 24756 leader_election.cc:288] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1684ee6e90fd424487d2e316d689569a (127.13.148.62:44525), 551afc33da5c4d4296020fbd4716f47f (127.13.148.60:42141)
I20260806 16:04:25.866096 24757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } }
I20260806 16:04:25.866737 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.866135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55908)
0806 16:04:25.866522 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.866623 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":79,"threads_started":1}
W20260806 16:04:25.866971 23665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
I20260806 16:04:25.867334 24757 leader_election.cc:288] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1684ee6e90fd424487d2e316d689569a (127.13.148.62:44525), d53ffdbc54aa4654ad2b573a5cb5125f (127.13.148.61:34099)
I20260806 16:04:25.867499 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.866721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55910)
0806 16:04:25.867328 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.867410 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":62,"threads_started":1}
W20260806 16:04:25.867676 23944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
I20260806 16:04:25.868170 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.867873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55916)
0806 16:04:25.868044 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.868100 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260806 16:04:25.868328 23665 leader_election.cc:334] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1684ee6e90fd424487d2e316d689569a (127.13.148.62:44525): Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
I20260806 16:04:25.868744 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.868451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55924)
0806 16:04:25.868645 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.868694 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 16:04:25.868849 23944 leader_election.cc:334] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1684ee6e90fd424487d2e316d689569a (127.13.148.62:44525): Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
I20260806 16:04:25.870712 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.866211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42141 (local address 127.0.0.1:34624)
0806 16:04:25.866575 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.866593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:25.866706 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:25.867025 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:25.867028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:25.867048 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:25.867157 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.867167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.868239 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.868243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.869593 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.869601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.870255 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.870258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.870366 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.870385 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:04:25.870501 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 16:04:25.870585 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":75,"threads_started":1}
I20260806 16:04:25.870724 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.866394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.13.148.60:42141)
0806 16:04:25.866623 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:25.866629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:25.866645 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:04:25.866837 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:25.866840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:25.866896 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:25.867012 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:25.867318 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.868103 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.869708 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.870128 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.870243 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.870304 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:25.870537 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 16:04:25.870610 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260806 16:04:25.871189 23987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "551afc33da5c4d4296020fbd4716f47f" is_pre_election: true
I20260806 16:04:25.871404 23987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53ffdbc54aa4654ad2b573a5cb5125f in term 1.
I20260806 16:04:25.871675 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.867152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34099 (local address 127.0.0.1:58302)
0806 16:04:25.867862 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.867875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:25.867963 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:25.868431 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:25.868433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:25.868450 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:25.868538 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.868543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.869571 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.869574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.870366 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.870375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.871256 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.871260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.871363 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.871380 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:04:25.871470 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:04:25.871539 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":54,"threads_started":1}
I20260806 16:04:25.871711 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.867431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.13.148.61:34099)
0806 16:04:25.868211 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:25.868221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:25.868231 (+ 10us) server_negotiation.cc:407] Connection header received
0806 16:04:25.868264 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:25.868266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:25.868311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:25.868639 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:25.868725 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.869457 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.870559 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.871104 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.871249 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.871302 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:25.871486 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 16:04:25.871558 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":104,"threads_started":1}
I20260806 16:04:25.871775 23662 leader_election.cc:302] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 551afc33da5c4d4296020fbd4716f47f, d53ffdbc54aa4654ad2b573a5cb5125f; no voters: 1684ee6e90fd424487d2e316d689569a
I20260806 16:04:25.872119 24756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 16:04:25.872188 24756 raft_consensus.cc:482] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1684ee6e90fd424487d2e316d689569a)
I20260806 16:04:25.872246 24756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:04:25.872287 23702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "551afc33da5c4d4296020fbd4716f47f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" is_pre_election: true
I20260806 16:04:25.873633 24756 raft_consensus.cc:504] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } }
I20260806 16:04:25.873905 23702 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 551afc33da5c4d4296020fbd4716f47f in current term 2: Already voted for candidate d53ffdbc54aa4654ad2b573a5cb5125f in this term.
I20260806 16:04:25.874090 24756 leader_election.cc:288] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [CANDIDATE]: Term 2 election: Requested vote from peers 1684ee6e90fd424487d2e316d689569a (127.13.148.62:44525), 551afc33da5c4d4296020fbd4716f47f (127.13.148.60:42141)
I20260806 16:04:25.874526 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.874245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55938)
0806 16:04:25.874347 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.874407 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 16:04:25.874583 23987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "551afc33da5c4d4296020fbd4716f47f"
I20260806 16:04:25.874729 23987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:04:25.874771 23940 leader_election.cc:302] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 551afc33da5c4d4296020fbd4716f47f; no voters: 1684ee6e90fd424487d2e316d689569a, d53ffdbc54aa4654ad2b573a5cb5125f
I20260806 16:04:25.875941 23987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53ffdbc54aa4654ad2b573a5cb5125f in term 2.
I20260806 16:04:25.876132 24757 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:04:25.876305 23662 leader_election.cc:302] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 551afc33da5c4d4296020fbd4716f47f, d53ffdbc54aa4654ad2b573a5cb5125f; no voters:
I20260806 16:04:25.876507 24756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:04:25.876744 24756 raft_consensus.cc:686] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [term 2 LEADER]: Becoming Leader. State: Replica: d53ffdbc54aa4654ad2b573a5cb5125f, State: Running, Role: LEADER
I20260806 16:04:25.876897 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.876648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55942)
0806 16:04:25.876743 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.876824 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 16:04:25.877018 23665 leader_election.cc:334] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1684ee6e90fd424487d2e316d689569a (127.13.148.62:44525): Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
I20260806 16:04:25.877041 24756 consensus_queue.cc:237] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } }
I20260806 16:04:25.878269 24767 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d53ffdbc54aa4654ad2b573a5cb5125f. Latest consensus state: current_term: 2 leader_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:25.878366 24767 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:25.878711 24772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:25.881058 24772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:25.881490 24772 catalog_manager.cc:1259] Loaded cluster ID: 21ed52bc8b6644e9ac2115ac33bbbcf9
I20260806 16:04:25.881578 24772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:25.882169 24772 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:25.882652 24772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Loaded TSK: 0
I20260806 16:04:25.883347 24772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:04:25.922837 23689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:25.926105 23689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:25.929646 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.929251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55954)
0806 16:04:25.929398 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.929496 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 16:04:25.929728 23987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [term 2 FOLLOWER]: Refusing update from remote peer d53ffdbc54aa4654ad2b573a5cb5125f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 16:04:25.930333 24767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [LEADER]: Connected to new peer: Peer: permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 16:04:25.930444 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.930199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.0.0.1:55968)
0806 16:04:25.930332 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.930386 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 16:04:25.930764 23665 consensus_peers.cc:487] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f -> Peer 1684ee6e90fd424487d2e316d689569a (127.13.148.62:44525): Couldn't send request to peer 1684ee6e90fd424487d2e316d689569a. Status: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 16:04:25.932256 24757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d53ffdbc54aa4654ad2b573a5cb5125f. Latest consensus state: current_term: 2 leader_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:25.932449 24757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:04:25.933614 24756 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:25.933786 24756 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:25.934010 24757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1684ee6e90fd424487d2e316d689569a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44525 } } peers { permanent_uuid: "d53ffdbc54aa4654ad2b573a5cb5125f" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34099 } attrs { promote: false } } peers { permanent_uuid: "551afc33da5c4d4296020fbd4716f47f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42141 } attrs { promote: false } } }
I20260806 16:04:25.934123 24757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551afc33da5c4d4296020fbd4716f47f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:04:25.945461 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.941120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:35219 (local address 127.0.0.1:33280)
0806 16:04:25.941317 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.941333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:25.941411 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:25.942118 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:25.942121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:25.942138 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:25.942239 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.942245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.943297 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.943303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.944180 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.944193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.945019 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.945022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.945121 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.945144 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:04:25.945244 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 16:04:25.945318 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":26}
I20260806 16:04:25.945463 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.941560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.13.148.3:35219)
0806 16:04:25.941832 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:25.941837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:25.941851 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:25.941906 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:25.941909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:25.941965 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:25.942361 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:25.942518 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.943190 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.944325 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.944891 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.945019 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.945090 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:25.945262 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 16:04:25.945333 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
I20260806 16:04:25.946000 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.940846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:40533 (local address 127.0.0.1:49420)
0806 16:04:25.941253 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.941271 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:25.941389 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:25.941817 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:25.941820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:25.941840 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:25.941974 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.941981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.943336 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.943340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.944631 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.944642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.945621 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.945625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.945721 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.945747 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 16:04:25.945798 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 16:04:25.945869 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316}
I20260806 16:04:25.946514 24547 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd3c91f452a46db9a689c81cde13d0c (DEFAULT_TABLE table=test-table [id=dc505e164bd243b799414eeafcef2eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:25.946672 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.940946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.13.148.2:40533)
0806 16:04:25.941525 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:25.941530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:25.941546 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:04:25.941603 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:25.941606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:25.941674 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:25.942169 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:25.942311 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.943194 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.944794 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.945485 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.946303 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.946377 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:25.946416 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 16:04:25.946521 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":147,"threads_started":1}
I20260806 16:04:25.946815 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.941887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:35813 (local address 127.0.0.1:60514)
0806 16:04:25.942165 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.942180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:25.942259 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:25.942839 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:25.942842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:25.942861 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:25.942941 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.942947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.944103 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.944107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.945346 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.945356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.946360 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.946364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:25.946451 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.946500 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 16:04:25.946587 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 16:04:25.946665 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
I20260806 16:04:25.946851 24547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd3c91f452a46db9a689c81cde13d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:25.947238 24346 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd3c91f452a46db9a689c81cde13d0c (DEFAULT_TABLE table=test-table [id=dc505e164bd243b799414eeafcef2eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:25.947580 24346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd3c91f452a46db9a689c81cde13d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:25.949286 24797 tablet_bootstrap.cc:492] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: Bootstrap starting.
I20260806 16:04:25.949973 24798 tablet_bootstrap.cc:492] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: Bootstrap starting.
I20260806 16:04:25.950169 24797 tablet_bootstrap.cc:654] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:25.950313 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.942255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.13.148.1:35813)
0806 16:04:25.942578 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:25.942583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:25.942600 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:04:25.942655 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:25.942658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:25.942707 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:25.942802 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:25.943082 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.943914 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.945521 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:25.946229 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:25.946343 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:25.946406 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:25.946617 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 16:04:25.946715 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260806 16:04:25.950874 24137 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd3c91f452a46db9a689c81cde13d0c (DEFAULT_TABLE table=test-table [id=dc505e164bd243b799414eeafcef2eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:25.951006 24798 tablet_bootstrap.cc:654] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:25.951200 24137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd3c91f452a46db9a689c81cde13d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:25.951658 24797 tablet_bootstrap.cc:492] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: No bootstrap required, opened a new log
I20260806 16:04:25.951756 24797 ts_tablet_manager.cc:1397] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:25.952234 24797 raft_consensus.cc:348] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:25.952335 24797 raft_consensus.cc:374] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:25.952358 24797 raft_consensus.cc:729] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f2160488f74534b5117515552567f7, State: Initialized, Role: FOLLOWER
I20260806 16:04:25.952493 24797 consensus_queue.cc:260] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:25.952884 24797 ts_tablet_manager.cc:1428] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:25.953722 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.953282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.3:55775)
0806 16:04:25.953551 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.953623 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":115,"threads_started":1}
I20260806 16:04:25.954414 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.954169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.3:57597)
0806 16:04:25.954273 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.954336 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 16:04:25.954663 24803 tablet_bootstrap.cc:492] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5: Bootstrap starting.
I20260806 16:04:25.955610 24803 tablet_bootstrap.cc:654] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:25.956172 24655 heartbeater.cc:503] Master 127.13.148.61:34099 was elected leader, sending a full tablet report...
I20260806 16:04:25.956993 24803 tablet_bootstrap.cc:492] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5: No bootstrap required, opened a new log
I20260806 16:04:25.956993 24798 tablet_bootstrap.cc:492] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: No bootstrap required, opened a new log
I20260806 16:04:25.957099 24803 ts_tablet_manager.cc:1397] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 16:04:25.957099 24798 ts_tablet_manager.cc:1397] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 16:04:25.957551 24803 raft_consensus.cc:348] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:25.957549 24798 raft_consensus.cc:348] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:25.957682 24798 raft_consensus.cc:374] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:25.957682 24803 raft_consensus.cc:374] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:25.957722 24798 raft_consensus.cc:729] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd0c89391ee4de4948cd407678ac1d0, State: Initialized, Role: FOLLOWER
I20260806 16:04:25.957721 24803 raft_consensus.cc:729] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42cbfbe63404bd39c4850e55be882f5, State: Initialized, Role: FOLLOWER
I20260806 16:04:25.957865 24798 consensus_queue.cc:260] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:25.957865 24803 consensus_queue.cc:260] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:25.958235 24798 ts_tablet_manager.cc:1428] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:25.958235 24803 ts_tablet_manager.cc:1428] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:25.959354 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.958948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.2:57721)
0806 16:04:25.959191 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.959247 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":124,"threads_started":1}
I20260806 16:04:25.959851 24448 heartbeater.cc:503] Master 127.13.148.61:34099 was elected leader, sending a full tablet report...
I20260806 16:04:25.961361 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.960269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.2:48321)
0806 16:04:25.961251 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.961308 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":899}
I20260806 16:04:25.961782 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.960717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.1:54881)
0806 16:04:25.961602 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.961657 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":104,"threads_started":1}
I20260806 16:04:25.961947 24237 heartbeater.cc:503] Master 127.13.148.61:34099 was elected leader, sending a full tablet report...
I20260806 16:04:25.996943 24801 raft_consensus.cc:482] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:25.997104 24801 raft_consensus.cc:504] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:25.998369 24801 leader_election.cc:288] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533), e42cbfbe63404bd39c4850e55be882f5 (127.13.148.1:35813)
I20260806 16:04:26.002918 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.998853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:57779 (local address 127.13.148.1:35813)
0806 16:04:25.999051 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:25.999055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:25.999165 (+ 110us) server_negotiation.cc:407] Connection header received
0806 16:04:25.999257 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:25.999262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:25.999313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:25.999379 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:25.999636 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.000438 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.001777 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.002192 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.002335 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:26.002392 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:26.002683 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 16:04:26.002777 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260806 16:04:26.002918 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.998567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:40533 (local address 127.13.148.3:42003)
0806 16:04:25.998758 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.998775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:25.998878 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:25.999167 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:25.999171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:25.999186 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:25.999323 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.999328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.000582 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.000588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:26.001585 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:26.001596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.002331 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.002335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:26.002491 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:26.002568 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 16:04:26.002675 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 16:04:26.002781 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":101}
I20260806 16:04:26.002929 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.998722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:42003 (local address 127.13.148.2:40533)
0806 16:04:25.998882 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:25.998888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:25.998907 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:04:25.998970 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:25.998973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:25.999051 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:25.999156 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:25.999470 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.000406 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.001733 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.002185 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.002328 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:26.002389 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:26.002714 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 16:04:26.002797 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:04:26.002965 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:25.998711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:35813 (local address 127.13.148.3:57779)
0806 16:04:25.999056 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:25.999071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:25.999167 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:25.999419 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:25.999422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:25.999444 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:25.999527 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:25.999531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.000584 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.000588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:26.001608 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:26.001620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.002340 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.002344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:26.002509 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:26.002569 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 16:04:26.002638 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 16:04:26.002768 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":105,"thread_start_us":147,"threads_started":1}
I20260806 16:04:26.003623 24371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd3c91f452a46db9a689c81cde13d0c" candidate_uuid: "52f2160488f74534b5117515552567f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd0c89391ee4de4948cd407678ac1d0" is_pre_election: true
I20260806 16:04:26.003846 24371 raft_consensus.cc:2454] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f2160488f74534b5117515552567f7 in term 0.
I20260806 16:04:26.003832 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd3c91f452a46db9a689c81cde13d0c" candidate_uuid: "52f2160488f74534b5117515552567f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42cbfbe63404bd39c4850e55be882f5" is_pre_election: true
I20260806 16:04:26.004112 24167 raft_consensus.cc:2454] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f2160488f74534b5117515552567f7 in term 0.
I20260806 16:04:26.004283 24484 leader_election.cc:302] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd0c89391ee4de4948cd407678ac1d0, 52f2160488f74534b5117515552567f7; no voters:
I20260806 16:04:26.004628 24801 raft_consensus.cc:2781] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:26.004763 24801 raft_consensus.cc:482] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:26.004873 24801 raft_consensus.cc:3037] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:26.006404 24801 raft_consensus.cc:504] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:26.006901 24801 leader_election.cc:288] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [CANDIDATE]: Term 1 election: Requested vote from peers 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533), e42cbfbe63404bd39c4850e55be882f5 (127.13.148.1:35813)
I20260806 16:04:26.007309 24167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd3c91f452a46db9a689c81cde13d0c" candidate_uuid: "52f2160488f74534b5117515552567f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42cbfbe63404bd39c4850e55be882f5"
I20260806 16:04:26.007453 24167 raft_consensus.cc:3037] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:26.008409 24371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd3c91f452a46db9a689c81cde13d0c" candidate_uuid: "52f2160488f74534b5117515552567f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd0c89391ee4de4948cd407678ac1d0"
I20260806 16:04:26.008574 24371 raft_consensus.cc:3037] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:26.008975 24167 raft_consensus.cc:2454] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f2160488f74534b5117515552567f7 in term 1.
I20260806 16:04:26.009337 24478 leader_election.cc:302] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f2160488f74534b5117515552567f7, e42cbfbe63404bd39c4850e55be882f5; no voters:
I20260806 16:04:26.009548 24801 raft_consensus.cc:2781] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:26.009773 24801 raft_consensus.cc:686] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 1 LEADER]: Becoming Leader. State: Replica: 52f2160488f74534b5117515552567f7, State: Running, Role: LEADER
I20260806 16:04:26.010073 24371 raft_consensus.cc:2454] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f2160488f74534b5117515552567f7 in term 1.
I20260806 16:04:26.010641 24801 consensus_queue.cc:237] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } }
I20260806 16:04:26.010855 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:26.010584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.3:33679)
0806 16:04:26.010702 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:26.010767 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 16:04:26.014837 23688 catalog_manager.cc:5495] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f2160488f74534b5117515552567f7 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "52f2160488f74534b5117515552567f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f2160488f74534b5117515552567f7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 } health_report { overall_health: UNKNOWN } } }
I20260806 16:04:26.045078 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:26.041590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.13.148.3:35219)
0806 16:04:26.041747 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:26.041751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:26.041766 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:04:26.041925 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:26.041927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:26.041983 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:26.042077 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:26.042346 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.042926 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.043774 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:26.043977 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:26.044056 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:26.044287 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:26.044734 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:26.044819 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:26.044912 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 16:04:26.044960 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:04:26.048307 24371 raft_consensus.cc:1261] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 1 FOLLOWER]: Refusing update from remote peer 52f2160488f74534b5117515552567f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:26.048496 24167 raft_consensus.cc:1261] T 0dd3c91f452a46db9a689c81cde13d0c P e42cbfbe63404bd39c4850e55be882f5 [term 1 FOLLOWER]: Refusing update from remote peer 52f2160488f74534b5117515552567f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:26.048899 24801 consensus_queue.cc:1035] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd0c89391ee4de4948cd407678ac1d0" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:26.049146 24818 consensus_queue.cc:1035] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e42cbfbe63404bd39c4850e55be882f5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:26.053565 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:26.050691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.2:47407)
0806 16:04:26.053402 (+ 2711us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:26.053468 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2610}
I20260806 16:04:26.054001 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:26.050762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.1:50755)
0806 16:04:26.053869 (+ 3107us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:26.053926 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3036}
I20260806 16:04:26.054740 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:26.054435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.3:51173)
0806 16:04:26.054616 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:26.054671 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260806 16:04:26.054986 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:26.054733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.1:39259)
0806 16:04:26.054872 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:26.054920 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 16:04:26.056682 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:26.056441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44525 (local address 127.13.148.2:44275)
0806 16:04:26.056534 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:26.056614 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260806 16:04:26.063090 23689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:26.063236 23689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:26.067158 23689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d53ffdbc54aa4654ad2b573a5cb5125f: Sending DeleteTablet for 3 replicas of tablet 0dd3c91f452a46db9a689c81cde13d0c
I20260806 16:04:26.067816 24346 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd3c91f452a46db9a689c81cde13d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:26 UTC) from {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:49420
I20260806 16:04:26.068179 24840 tablet_replica.cc:330] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: stopping tablet replica
I20260806 16:04:26.068264 24547 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd3c91f452a46db9a689c81cde13d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:26 UTC) from {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:33280
I20260806 16:04:26.068325 24840 raft_consensus.cc:2229] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:26.068463 24840 raft_consensus.cc:2258] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:26.068974 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 23899
I20260806 16:04:26.069010 24137 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd3c91f452a46db9a689c81cde13d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:26 UTC) from {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:60514
I20260806 16:04:26.069240 24840 ts_tablet_manager.cc:1905] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:26.069684 24841 tablet_replica.cc:330] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: stopping tablet replica
I20260806 16:04:26.069811 24841 raft_consensus.cc:2229] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:26.070070 24841 raft_consensus.cc:2258] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:26.072142 24841 ts_tablet_manager.cc:1905] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:26.072275 24840 ts_tablet_manager.cc:1918] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:26.072341 24840 log.cc:1196] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/0dd3c91f452a46db9a689c81cde13d0c
I20260806 16:04:26.072645 24840 ts_tablet_manager.cc:1939] T 0dd3c91f452a46db9a689c81cde13d0c P 1cd0c89391ee4de4948cd407678ac1d0: Deleting consensus metadata
I20260806 16:04:26.073309 23665 catalog_manager.cc:4841] TS 1cd0c89391ee4de4948cd407678ac1d0 (127.13.148.2:40533): tablet 0dd3c91f452a46db9a689c81cde13d0c (table test-table [id=dc505e164bd243b799414eeafcef2eaa]) successfully deleted
I20260806 16:04:26.074733 24841 ts_tablet_manager.cc:1918] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:26.074872 24841 log.cc:1196] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/0dd3c91f452a46db9a689c81cde13d0c
I20260806 16:04:26.075132 24841 ts_tablet_manager.cc:1939] T 0dd3c91f452a46db9a689c81cde13d0c P 52f2160488f74534b5117515552567f7: Deleting consensus metadata
I20260806 16:04:26.075775 23663 catalog_manager.cc:4841] TS 52f2160488f74534b5117515552567f7 (127.13.148.3:35219): tablet 0dd3c91f452a46db9a689c81cde13d0c (table test-table [id=dc505e164bd243b799414eeafcef2eaa]) successfully deleted
I20260806 16:04:26.084913 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 24251
I20260806 16:04:26.085096 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:26.084796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:35813 (local address 127.0.0.1:60530)
0806 16:04:26.084916 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:26.084997 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:35813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 16:04:26.087021 23662 catalog_manager.cc:4584] TS e42cbfbe63404bd39c4850e55be882f5 (127.13.148.1:35813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0dd3c91f452a46db9a689c81cde13d0c: Network error: Client connection negotiation failed: client connection to 127.13.148.1:35813: connect: Connection refused (error 111)
I20260806 16:04:26.097741 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 24453
I20260806 16:04:26.109843 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 23480
I20260806 16:04:26.123768 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 23646
2026-08-06T16:04:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5731 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:26 dist-test-slave-7kzw krb5kdc[24845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:26 dist-test-slave-7kzw krb5kdc[24845](info): set up 2 sockets
Aug 06 16:04:26 dist-test-slave-7kzw krb5kdc[24845](info): commencing operation
krb5kdc: starting...
W20260806 16:04:27.377272 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:27 dist-test-slave-7kzw krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.62.keytab.
Entry for principal oryx/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.62.keytab.
I20260806 16:04:27.510382 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44803
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.62.keytab
--principal=oryx/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:44803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:27.625322 24866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:27.625587 24866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:27.625628 24866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:27.629769 24866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:27.629825 24866 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 16:04:27.629843 24866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:27.629887 24866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:27.629925 24866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:27.629945 24866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:27.635478 24866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:44803
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.62.keytab
--principal=oryx/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:27.636554 24866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:27.637712 24866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:27.646162 24881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:27.646162 24877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:27.646202 24878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:27.646224 24866 server_base.cc:1034] running on GCE node
I20260806 16:04:27.647089 24866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:27.648170 24866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:27.649384 24866 hybrid_clock.cc:648] HybridClock initialized: now 1786032267649333 us; error 73 us; skew 500 ppm
Aug 06 16:04:27 dist-test-slave-7kzw krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032267, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:27.652326 24866 init.cc:377] Logged in from keytab as oryx/127.13.148.62@KRBTEST.COM (short username oryx)
I20260806 16:04:27.653301 24866 webserver.cc:460] Webserver started at http://127.13.148.62:40773/ using document root <none> and password file <none>
I20260806 16:04:27.653894 24866 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:27.653990 24866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:27.654245 24866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:27.656433 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "3927ff48d6814166b137039c8a48dc38"
format_stamp: "Formatted at 2026-08-06 16:04:27 on dist-test-slave-7kzw"
server_key: "1e64bff7f9f62e696c7d194c3ce3a4a6"
server_key_iv: "c400d1ace1dc72da1209ebf380067045"
server_key_version: "encryptionkey@0"
I20260806 16:04:27.657043 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "3927ff48d6814166b137039c8a48dc38"
format_stamp: "Formatted at 2026-08-06 16:04:27 on dist-test-slave-7kzw"
server_key: "1e64bff7f9f62e696c7d194c3ce3a4a6"
server_key_iv: "c400d1ace1dc72da1209ebf380067045"
server_key_version: "encryptionkey@0"
I20260806 16:04:27.661052 24866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 16:04:27.663780 24894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:27.665637 24866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:04:27.665786 24866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "3927ff48d6814166b137039c8a48dc38"
format_stamp: "Formatted at 2026-08-06 16:04:27 on dist-test-slave-7kzw"
server_key: "1e64bff7f9f62e696c7d194c3ce3a4a6"
server_key_iv: "c400d1ace1dc72da1209ebf380067045"
server_key_version: "encryptionkey@0"
I20260806 16:04:27.665934 24866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:27.686187 24866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:27.689632 24866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:27.689843 24866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:27.698350 24866 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:44803
I20260806 16:04:27.698365 24993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:44803 every 8 connection(s)
I20260806 16:04:27.699409 24866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:27.702250 24994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:27.707960 24994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: Bootstrap starting.
I20260806 16:04:27.708267 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 24866
I20260806 16:04:27.708365 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:27.708729 13904 external_mini_cluster.cc:1428] Setting key 344e95ddd3dc04434657336616c98e8c
I20260806 16:04:27.710759 24994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:27.711516 24994 log.cc:824] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:27.713676 24994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: No bootstrap required, opened a new log
Aug 06 16:04:27 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:27.718350 24994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927ff48d6814166b137039c8a48dc38" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44803 } }
I20260806 16:04:27.718674 24994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:27.718768 24994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3927ff48d6814166b137039c8a48dc38, State: Initialized, Role: FOLLOWER
I20260806 16:04:27.719262 24994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927ff48d6814166b137039c8a48dc38" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44803 } }
I20260806 16:04:27.719417 24994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:27.719509 24994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:27.719609 24994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:27.720539 24994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927ff48d6814166b137039c8a48dc38" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44803 } }
I20260806 16:04:27.720930 24994 leader_election.cc:302] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3927ff48d6814166b137039c8a48dc38; no voters:
I20260806 16:04:27.721318 24994 leader_election.cc:288] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:27.721419 24999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:27.721607 24999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [term 1 LEADER]: Becoming Leader. State: Replica: 3927ff48d6814166b137039c8a48dc38, State: Running, Role: LEADER
I20260806 16:04:27.721901 24999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927ff48d6814166b137039c8a48dc38" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44803 } }
I20260806 16:04:27.722661 24994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:27.723076 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3927ff48d6814166b137039c8a48dc38. Latest consensus state: current_term: 1 leader_uuid: "3927ff48d6814166b137039c8a48dc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927ff48d6814166b137039c8a48dc38" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44803 } } }
I20260806 16:04:27.723203 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:27.723615 25005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:27.723771 25002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3927ff48d6814166b137039c8a48dc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927ff48d6814166b137039c8a48dc38" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44803 } } }
I20260806 16:04:27.723953 25002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:27.724670 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:27.710050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.13.148.62:44803)
0806 16:04:27.710460 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:27.710505 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:27.710555 (+ 50us) server_negotiation.cc:407] Connection header received
0806 16:04:27.711273 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:27.711293 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:27.711625 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:27.711979 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:27.712878 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:27.713927 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:27.715115 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:27.715573 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:27.715696 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:27.718629 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:27.718634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:27.718648 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:27.718702 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:27.720627 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:27.721257 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:27.721264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:27.721270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:27.721435 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:27.721796 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:27.721799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:27.721801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:27.722262 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:27.722507 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:27.723154 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 16:04:27.723504 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":148,"threads_started":1}
I20260806 16:04:27.726992 25005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:27.733131 25005 catalog_manager.cc:1347] Generated new cluster ID: ddbfc1f533b04b219b98bfda67bde23a
I20260806 16:04:27.733217 25005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:27.757145 25005 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:27.758220 25005 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:27.769109 25005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: Generated new TSK 0
I20260806 16:04:27.769899 25005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
I20260806 16:04:27.806259 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44803
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:27.946313 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:27.946586 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:27.946630 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:27.950611 25032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:27.950672 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:27.950711 25032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:04:27.950783 25032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:27.955927 25032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:27.956959 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:27.957976 25032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:27.967845 25043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:27.967808 25047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:27.967969 25044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:27.967868 25032 server_base.cc:1034] running on GCE node
I20260806 16:04:27.968535 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:27.969197 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:27.970396 25032 hybrid_clock.cc:648] HybridClock initialized: now 1786032267970373 us; error 31 us; skew 500 ppm
Aug 06 16:04:27 dist-test-slave-7kzw krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032267, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:27.973178 25032 init.cc:377] Logged in from keytab as oryx/127.13.148.1@KRBTEST.COM (short username oryx)
I20260806 16:04:27.974016 25032 webserver.cc:460] Webserver started at http://127.13.148.1:39671/ using document root <none> and password file <none>
I20260806 16:04:27.974653 25032 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:27.974757 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:27.974995 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:27.977001 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "4f60f689a5c74e05ad2d2ca43fcea86d"
format_stamp: "Formatted at 2026-08-06 16:04:27 on dist-test-slave-7kzw"
server_key: "1df58cb3f280d208be2206bdcdb0f446"
server_key_iv: "c1ae9d449a90950fe25eaac3fe246f34"
server_key_version: "encryptionkey@0"
I20260806 16:04:27.977596 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "4f60f689a5c74e05ad2d2ca43fcea86d"
format_stamp: "Formatted at 2026-08-06 16:04:27 on dist-test-slave-7kzw"
server_key: "1df58cb3f280d208be2206bdcdb0f446"
server_key_iv: "c1ae9d449a90950fe25eaac3fe246f34"
server_key_version: "encryptionkey@0"
I20260806 16:04:27.981523 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 16:04:27.984076 25058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:27.985509 25032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:04:27.985690 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "4f60f689a5c74e05ad2d2ca43fcea86d"
format_stamp: "Formatted at 2026-08-06 16:04:27 on dist-test-slave-7kzw"
server_key: "1df58cb3f280d208be2206bdcdb0f446"
server_key_iv: "c1ae9d449a90950fe25eaac3fe246f34"
server_key_version: "encryptionkey@0"
I20260806 16:04:27.985821 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:28.009054 25032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:28.012526 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:28.012771 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:28.014336 25032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:28.014422 25032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:28.014529 25032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:28.014599 25032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032267, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:28.027057 25032 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:34033
I20260806 16:04:28.027132 25233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:34033 every 8 connection(s)
I20260806 16:04:28.028064 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:28.029706 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.015357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:59857 (local address 127.13.148.62:44803)
0806 16:04:28.015547 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.015555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.016332 (+ 777us) server_negotiation.cc:407] Connection header received
0806 16:04:28.017286 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.017293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.017362 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.017477 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:28.018972 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.019635 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.020800 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.021093 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.021237 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.025546 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:28.025550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:28.025553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:28.025584 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:28.027422 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.027952 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.027956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.027958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.028053 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.028436 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.028441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.028443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.028718 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:28.028820 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.029432 (+ 612us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.029550 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:04:28.030403 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.015694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44803 (local address 127.13.148.1:59857)
0806 16:04:28.016151 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.016203 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.017065 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.017679 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.017688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.018211 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:28.018790 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.018812 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.019771 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.019775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.020667 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.020677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.021286 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.021293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.021563 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.022279 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:28.022300 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:28.025346 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:28.027545 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.027551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.027567 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.027841 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.028160 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.028163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.028165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.028334 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.028876 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:28.028884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:28.029176 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.029339 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.029584 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":130,"threads_started":1}
I20260806 16:04:28.031818 25234 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44803
I20260806 16:04:28.031989 25234 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:28.032544 25234 heartbeater.cc:511] Master 127.13.148.62:44803 requested a full tablet report, sending...
I20260806 16:04:28.034792 24920 ts_manager.cc:194] Registered new tserver with Master: 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033)
I20260806 16:04:28.035032 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 25032
I20260806 16:04:28.035154 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:28.035485 13904 external_mini_cluster.cc:1428] Setting key 37dfa699d8aaf82294082c97e79ade6c
I20260806 16:04:28.036330 24920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:59857
I20260806 16:04:28.043233 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.034225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44803 (local address 127.13.148.1:48399)
0806 16:04:28.034643 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.034675 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.034800 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.035050 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.035053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.035280 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:28.035393 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.035398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.036884 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.036886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.037713 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.037724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.038384 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.038388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.038505 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.039103 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:28.039123 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:28.039614 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:28.041357 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.041363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.041368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.041687 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.042092 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.042098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.042101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.042216 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.042680 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:28.042683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:28.042770 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.042885 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.043038 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":97,"threads_started":1}
I20260806 16:04:28.043227 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.034317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48399 (local address 127.13.148.62:44803)
0806 16:04:28.034438 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.034441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.034746 (+ 305us) server_negotiation.cc:407] Connection header received
0806 16:04:28.034901 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.034904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.034948 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.035034 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:28.035508 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.036731 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.037849 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.038250 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.038373 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.039769 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:28.039772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:28.039774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:28.039796 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:28.041241 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.041847 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.041850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.041851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.041976 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.042315 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.042317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.042318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.042542 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:28.042655 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.042940 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.043054 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":584,"server-negotiator.queue_time_us":41}
WARNING: no policy specified for oryx/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
I20260806 16:04:28.066707 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44803
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:28.179894 25247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:28.180186 25247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:28.180258 25247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:28.184286 25247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:28.184370 25247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:28.184437 25247 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:04:28.184520 25247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:28.189805 25247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:28.191020 25247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:28.192080 25247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:28.201819 25261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:28.201819 25256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:28.201959 25257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:28.202904 25247 server_base.cc:1034] running on GCE node
I20260806 16:04:28.203337 25247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:28.204010 25247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:28.205194 25247 hybrid_clock.cc:648] HybridClock initialized: now 1786032268205179 us; error 29 us; skew 500 ppm
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032268, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:28.208163 25247 init.cc:377] Logged in from keytab as oryx/127.13.148.2@KRBTEST.COM (short username oryx)
I20260806 16:04:28.209092 25247 webserver.cc:460] Webserver started at http://127.13.148.2:33721/ using document root <none> and password file <none>
I20260806 16:04:28.209676 25247 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:28.209774 25247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:28.210011 25247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:28.212078 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "6baca6740ec549559979bcf238b1c292"
format_stamp: "Formatted at 2026-08-06 16:04:28 on dist-test-slave-7kzw"
server_key: "489bed8ff641a04c686e03fcd60b4c07"
server_key_iv: "b077b948934d6d565e16084dc256f127"
server_key_version: "encryptionkey@0"
I20260806 16:04:28.212658 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "6baca6740ec549559979bcf238b1c292"
format_stamp: "Formatted at 2026-08-06 16:04:28 on dist-test-slave-7kzw"
server_key: "489bed8ff641a04c686e03fcd60b4c07"
server_key_iv: "b077b948934d6d565e16084dc256f127"
server_key_version: "encryptionkey@0"
I20260806 16:04:28.216439 25247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:04:28.218989 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:28.220323 25247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:28.220482 25247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "6baca6740ec549559979bcf238b1c292"
format_stamp: "Formatted at 2026-08-06 16:04:28 on dist-test-slave-7kzw"
server_key: "489bed8ff641a04c686e03fcd60b4c07"
server_key_iv: "b077b948934d6d565e16084dc256f127"
server_key_version: "encryptionkey@0"
I20260806 16:04:28.220599 25247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:28.267158 25247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:28.270988 25247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:28.271207 25247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:28.272786 25247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:28.272867 25247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:28.272943 25247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:28.272995 25247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032268, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:28.284083 25247 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:39383
I20260806 16:04:28.285104 25247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:28.285841 25449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:39383 every 8 connection(s)
I20260806 16:04:28.288472 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.273832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:55641 (local address 127.13.148.62:44803)
0806 16:04:28.273966 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.273970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.275113 (+ 1143us) server_negotiation.cc:407] Connection header received
0806 16:04:28.276361 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.276364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.276426 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.276549 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:28.278250 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.279211 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.280693 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.281067 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.281207 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.284097 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:28.284101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:28.284104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:28.284144 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:28.285978 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.286750 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.286754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.286756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.286861 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.287248 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.287251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.287253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.287474 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:28.287601 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.288145 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.288282 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:04:28.289311 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.274350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44803 (local address 127.13.148.2:55641)
0806 16:04:28.274922 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.274969 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.276128 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.276750 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.276768 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.277260 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:28.278082 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.278106 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.279342 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.279353 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.280544 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.280555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.281251 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.281256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.281460 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.282162 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:28.282183 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:28.283907 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:28.286117 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.286124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.286139 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.286632 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.286966 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.286971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.286973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.287132 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.287612 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:28.287621 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:28.287874 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.288079 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.288411 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":139,"threads_started":1}
I20260806 16:04:28.290709 25451 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44803
I20260806 16:04:28.290871 25451 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:28.291378 25451 heartbeater.cc:511] Master 127.13.148.62:44803 requested a full tablet report, sending...
I20260806 16:04:28.292488 24922 ts_manager.cc:194] Registered new tserver with Master: 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383)
I20260806 16:04:28.293147 24922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:55641
I20260806 16:04:28.294930 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 25247
I20260806 16:04:28.295053 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:28.295403 13904 external_mini_cluster.cc:1428] Setting key 62b1c7a5dc6b8a66424429d6fc21662d
I20260806 16:04:28.301479 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.293377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:57935 (local address 127.13.148.62:44803)
0806 16:04:28.293511 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.293516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.293607 (+ 91us) server_negotiation.cc:407] Connection header received
0806 16:04:28.293714 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.293717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.293765 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.293860 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:28.294332 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.295178 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.296095 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.296353 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.296438 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.297632 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:28.297638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:28.297645 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:28.297686 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:28.299601 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.300159 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.300161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.300163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.300243 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.300613 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.300615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.300616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.300829 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:28.300915 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.301259 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.301360 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:04:28.301637 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.293282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44803 (local address 127.13.148.2:57935)
0806 16:04:28.293542 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.293554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.293634 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.293875 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.293878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.294101 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:28.294204 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.294210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.295311 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.295314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.295985 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.295995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.296475 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.296481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.296586 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.297133 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:28.297144 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:28.297466 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:28.299726 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.299732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.299736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.300020 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.300339 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.300345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.300347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.300458 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.300943 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:28.300949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:28.301065 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.301195 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.301363 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
I20260806 16:04:28.326179 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44803
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:28.443925 25461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:28.444182 25461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:28.444229 25461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:28.448472 25461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:28.448539 25461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:28.448578 25461 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 16:04:28.448611 25461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:28.453904 25461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:44803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:28.455092 25461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:28.456205 25461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:28.466672 25467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:28.466713 25468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:28.466753 25470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:28.467068 25461 server_base.cc:1034] running on GCE node
I20260806 16:04:28.467490 25461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:28.468180 25461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:28.469384 25461 hybrid_clock.cc:648] HybridClock initialized: now 1786032268469359 us; error 35 us; skew 500 ppm
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032268, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:28.472371 25461 init.cc:377] Logged in from keytab as oryx/127.13.148.3@KRBTEST.COM (short username oryx)
I20260806 16:04:28.473309 25461 webserver.cc:460] Webserver started at http://127.13.148.3:33271/ using document root <none> and password file <none>
I20260806 16:04:28.473883 25461 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:28.473978 25461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:28.474227 25461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:28.476488 25461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "46e9679496e3430590d9cba787e88ccb"
format_stamp: "Formatted at 2026-08-06 16:04:28 on dist-test-slave-7kzw"
server_key: "2e994ac9bde194af903a40c65643c35a"
server_key_iv: "1629d800dee84be2b8bee271e501e039"
server_key_version: "encryptionkey@0"
I20260806 16:04:28.477350 25461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "46e9679496e3430590d9cba787e88ccb"
format_stamp: "Formatted at 2026-08-06 16:04:28 on dist-test-slave-7kzw"
server_key: "2e994ac9bde194af903a40c65643c35a"
server_key_iv: "1629d800dee84be2b8bee271e501e039"
server_key_version: "encryptionkey@0"
I20260806 16:04:28.481401 25461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:04:28.484479 25481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:28.485740 25461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:04:28.485908 25461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "46e9679496e3430590d9cba787e88ccb"
format_stamp: "Formatted at 2026-08-06 16:04:28 on dist-test-slave-7kzw"
server_key: "2e994ac9bde194af903a40c65643c35a"
server_key_iv: "1629d800dee84be2b8bee271e501e039"
server_key_version: "encryptionkey@0"
I20260806 16:04:28.486035 25461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:28.496413 25461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:28.499712 25461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:28.499926 25461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:28.501487 25461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:28.501567 25461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:28.501642 25461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:28.501708 25461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032268, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:28.514441 25461 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36783
I20260806 16:04:28.514542 25652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36783 every 8 connection(s)
I20260806 16:04:28.515487 25461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:28.517413 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.502199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:59399 (local address 127.13.148.62:44803)
0806 16:04:28.502360 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.502366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.503415 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 16:04:28.504781 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.504784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.504846 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.504946 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:28.506445 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.507085 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.508471 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.508924 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.509076 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.513315 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:28.513321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:28.513326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:28.513355 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:28.515139 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.515688 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.515692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.515694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.515798 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.516318 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.516322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.516324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.516569 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:28.516668 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.517171 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.517282 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:04:28.518285 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.502716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44803 (local address 127.13.148.3:59399)
0806 16:04:28.503226 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.503289 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.504520 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.505352 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.505361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.505783 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:28.506281 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.506296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.507234 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.507238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.508292 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.508304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.509107 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.509114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.509376 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.510026 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:28.510055 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:28.513101 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:28.515269 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.515275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.515290 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.515557 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.515967 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.515971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.515973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.516170 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.516699 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:28.516705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:28.516939 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.517108 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.517363 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
I20260806 16:04:28.519568 25653 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44803
I20260806 16:04:28.519785 25653 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:28.520341 25653 heartbeater.cc:511] Master 127.13.148.62:44803 requested a full tablet report, sending...
I20260806 16:04:28.521569 24922 ts_manager.cc:194] Registered new tserver with Master: 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:28.522472 24922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:59399
I20260806 16:04:28.523645 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 25461
I20260806 16:04:28.523780 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:28.524180 13904 external_mini_cluster.cc:1428] Setting key 04b360e397cbbe85ba106aec7c69e970
I20260806 16:04:28.526338 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:04:28.530627 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.521924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50225 (local address 127.13.148.62:44803)
0806 16:04:28.522044 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.522049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.522196 (+ 147us) server_negotiation.cc:407] Connection header received
0806 16:04:28.522280 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.522283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.522327 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.522405 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:28.523023 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.523930 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.525774 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.525995 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.526115 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.527226 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:28.527229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:28.527232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:28.527257 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:28.528657 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.529244 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.529248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.529250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.529350 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.529688 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.529690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.529691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.529863 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:28.529947 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.530307 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.530412 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:04:28.530659 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.521820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44803 (local address 127.13.148.3:50225)
0806 16:04:28.522107 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.522120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.522201 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.522510 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.522516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.522734 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:28.522897 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.522905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.524886 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.524889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.525644 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.525653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.526117 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.526121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.526214 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.526750 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:28.526761 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:28.527086 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:28.528772 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.528778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.528783 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.529073 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.529467 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:28.529471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:28.529474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:28.529581 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:28.529964 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:28.529968 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:28.530081 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.530211 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.530375 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032268, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:04:28.555681 24997 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:28.548256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.13.148.62:44803)
0806 16:04:28.548403 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.548406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.548419 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:04:28.548458 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.548460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.548498 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.548713 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:28.549761 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.550613 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.551590 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.551917 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.552032 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.555522 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48962: BlockingRecv error: recv got EOF from 127.0.0.1:48962 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 16:04:28 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032268, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:28.566039 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.557246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.13.148.62:44803)
0806 16:04:28.557385 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.557389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.557471 (+ 82us) server_negotiation.cc:407] Connection header received
0806 16:04:28.557555 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.557557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.557593 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.557662 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:28.558382 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.558956 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.559760 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.560024 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.560115 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.562432 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:28.562435 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:28.562437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:28.562508 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:28.564107 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.564616 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.564622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.564624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.564765 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:28.565100 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:28.565104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:28.565106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:28.565305 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:28.565455 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.565727 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.565856 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:04:28.569311 24922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:28.572602 24922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:28.587901 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.581599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39383 (local address 127.0.0.1:50486)
0806 16:04:28.582368 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.582426 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.582598 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.583296 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.583305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.583339 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:28.583532 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.583538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.584781 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.584788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.586109 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.586119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.587314 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.587318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.587439 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.587482 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.587603 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.587731 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"spinlock_wait_cycles":5376,"thread_start_us":102,"threads_started":1}
I20260806 16:04:28.587944 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.581779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.13.148.2:39383)
0806 16:04:28.582303 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.582316 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.582779 (+ 463us) server_negotiation.cc:407] Connection header received
0806 16:04:28.582870 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.582876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.583100 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.583280 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:28.583668 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.584656 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.586254 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.587157 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.587294 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.587474 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.587666 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.587770 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":135,"threads_started":1}
I20260806 16:04:28.588428 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.581658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36783 (local address 127.0.0.1:43800)
0806 16:04:28.583282 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.583297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.583406 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.583941 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.583945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.583970 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:28.584060 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.584066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.585174 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.585178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.586521 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.586534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.587930 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.587937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.588049 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.588078 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.588203 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.588284 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"spinlock_wait_cycles":896,"thread_start_us":93,"threads_started":1}
I20260806 16:04:28.588774 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.582621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.13.148.3:36783)
0806 16:04:28.583074 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.583085 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.583391 (+ 306us) server_negotiation.cc:407] Connection header received
0806 16:04:28.583560 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.583569 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.583786 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.583944 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:28.584192 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.585029 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.586657 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.587551 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.587687 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.587860 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.588239 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.588624 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":140,"threads_started":1}
I20260806 16:04:28.589785 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.582391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34033 (local address 127.0.0.1:52746)
0806 16:04:28.583546 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.583561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.583658 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.585192 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.585195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.585216 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:28.585306 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.585312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.586825 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.586832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.587914 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.587929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.589377 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.589381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.589482 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.589507 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.589584 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.589653 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"thread_start_us":140,"threads_started":1}
I20260806 16:04:28.590211 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.582839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.13.148.1:34033)
0806 16:04:28.583314 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.583321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.583758 (+ 437us) server_negotiation.cc:407] Connection header received
0806 16:04:28.583845 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.583854 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.584038 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.584230 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:28.585449 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.586707 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.588179 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.589037 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.589146 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.589302 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.589967 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.590067 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":102,"threads_started":1}
I20260806 16:04:28.591768 25353 tablet_service.cc:1467] Processing CreateTablet for tablet e846961e79b842c993ee98c651c564df (DEFAULT_TABLE table=test-table [id=2d4fcc86f43a4b02b6454dd53a90027c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:28.591861 25553 tablet_service.cc:1467] Processing CreateTablet for tablet e846961e79b842c993ee98c651c564df (DEFAULT_TABLE table=test-table [id=2d4fcc86f43a4b02b6454dd53a90027c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:28.592731 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e846961e79b842c993ee98c651c564df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:28.592947 25553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e846961e79b842c993ee98c651c564df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:28.593312 25130 tablet_service.cc:1467] Processing CreateTablet for tablet e846961e79b842c993ee98c651c564df (DEFAULT_TABLE table=test-table [id=2d4fcc86f43a4b02b6454dd53a90027c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:28.594352 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e846961e79b842c993ee98c651c564df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:28.600804 25688 tablet_bootstrap.cc:492] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: Bootstrap starting.
I20260806 16:04:28.602309 25689 tablet_bootstrap.cc:492] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: Bootstrap starting.
I20260806 16:04:28.602834 25692 tablet_bootstrap.cc:492] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: Bootstrap starting.
I20260806 16:04:28.603693 25688 tablet_bootstrap.cc:654] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:28.604516 25689 tablet_bootstrap.cc:654] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:28.604624 25688 log.cc:824] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:28.604920 25692 tablet_bootstrap.cc:654] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:28.605350 25689 log.cc:824] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:28.605818 25692 log.cc:824] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:28.606992 25688 tablet_bootstrap.cc:492] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: No bootstrap required, opened a new log
I20260806 16:04:28.607221 25688 ts_tablet_manager.cc:1397] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:04:28.607518 25689 tablet_bootstrap.cc:492] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: No bootstrap required, opened a new log
I20260806 16:04:28.607719 25689 ts_tablet_manager.cc:1397] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:04:28.608112 25692 tablet_bootstrap.cc:492] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: No bootstrap required, opened a new log
I20260806 16:04:28.608326 25692 ts_tablet_manager.cc:1397] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:04:28.611691 25688 raft_consensus.cc:348] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.611995 25688 raft_consensus.cc:374] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:28.612056 25688 raft_consensus.cc:729] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6baca6740ec549559979bcf238b1c292, State: Initialized, Role: FOLLOWER
I20260806 16:04:28.611953 25689 raft_consensus.cc:348] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.612205 25689 raft_consensus.cc:374] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:28.612279 25689 raft_consensus.cc:729] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9679496e3430590d9cba787e88ccb, State: Initialized, Role: FOLLOWER
I20260806 16:04:28.612228 25692 raft_consensus.cc:348] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.612524 25692 raft_consensus.cc:374] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:28.612598 25692 raft_consensus.cc:729] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f60f689a5c74e05ad2d2ca43fcea86d, State: Initialized, Role: FOLLOWER
I20260806 16:04:28.612593 25688 consensus_queue.cc:260] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.612673 25689 consensus_queue.cc:260] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.613042 25692 consensus_queue.cc:260] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.613400 25451 heartbeater.cc:503] Master 127.13.148.62:44803 was elected leader, sending a full tablet report...
I20260806 16:04:28.613731 25689 ts_tablet_manager.cc:1428] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:04:28.614035 25692 ts_tablet_manager.cc:1428] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:04:28.614094 25653 heartbeater.cc:503] Master 127.13.148.62:44803 was elected leader, sending a full tablet report...
I20260806 16:04:28.614539 25234 heartbeater.cc:503] Master 127.13.148.62:44803 was elected leader, sending a full tablet report...
I20260806 16:04:28.615804 25688 ts_tablet_manager.cc:1428] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 16:04:28.678432 25702 raft_consensus.cc:482] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:28.678731 25702 raft_consensus.cc:504] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.680035 25702 leader_election.cc:288] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383), 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:28.686146 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.680538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:36843 (local address 127.13.148.3:36783)
0806 16:04:28.680865 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.680871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.680891 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:04:28.680944 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.680946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.681008 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.681089 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:28.681584 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.682518 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.684702 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.685356 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.685495 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.685555 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.685916 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.685997 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260806 16:04:28.686162 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.680424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:47095 (local address 127.13.148.2:39383)
0806 16:04:28.680586 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.680591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.680780 (+ 189us) server_negotiation.cc:407] Connection header received
0806 16:04:28.680947 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.680951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.681001 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.681089 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:28.681583 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.682518 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.684624 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.685357 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.685495 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.685555 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.685916 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.685997 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 16:04:28.686244 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.680259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39383 (local address 127.13.148.1:47095)
0806 16:04:28.680684 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.680698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.680830 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.681135 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.681139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.681170 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:28.681418 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.681426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.682674 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.682677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.684421 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.684435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.685513 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.685518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.685737 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.685792 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.685879 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.686026 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":317,"spinlock_wait_cycles":896,"thread_start_us":147,"threads_started":1}
I20260806 16:04:28.686251 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.680411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36783 (local address 127.13.148.1:36843)
0806 16:04:28.680710 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:28.680723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:28.680827 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:28.681148 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:28.681150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:28.681175 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:28.681418 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.681426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.682665 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.682669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.684426 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:28.684435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.685513 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.685518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:28.685715 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.685786 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 16:04:28.685879 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 16:04:28.686030 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":310,"spinlock_wait_cycles":3840,"thread_start_us":130,"threads_started":1}
I20260806 16:04:28.686925 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e846961e79b842c993ee98c651c564df" candidate_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6baca6740ec549559979bcf238b1c292" is_pre_election: true
I20260806 16:04:28.687306 25375 raft_consensus.cc:2454] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f60f689a5c74e05ad2d2ca43fcea86d in term 0.
I20260806 16:04:28.687321 25577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e846961e79b842c993ee98c651c564df" candidate_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9679496e3430590d9cba787e88ccb" is_pre_election: true
I20260806 16:04:28.687605 25577 raft_consensus.cc:2454] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f60f689a5c74e05ad2d2ca43fcea86d in term 0.
I20260806 16:04:28.687881 25062 leader_election.cc:302] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f60f689a5c74e05ad2d2ca43fcea86d, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:28.688179 25702 raft_consensus.cc:2781] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:28.688294 25702 raft_consensus.cc:482] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:28.688364 25702 raft_consensus.cc:3037] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:28.689707 25702 raft_consensus.cc:504] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.690146 25702 leader_election.cc:288] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [CANDIDATE]: Term 1 election: Requested vote from peers 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383), 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:28.690586 25577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e846961e79b842c993ee98c651c564df" candidate_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9679496e3430590d9cba787e88ccb"
I20260806 16:04:28.690748 25577 raft_consensus.cc:3037] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:28.690866 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e846961e79b842c993ee98c651c564df" candidate_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6baca6740ec549559979bcf238b1c292"
I20260806 16:04:28.690977 25375 raft_consensus.cc:3037] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:28.692144 25375 raft_consensus.cc:2454] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f60f689a5c74e05ad2d2ca43fcea86d in term 1.
I20260806 16:04:28.692349 25577 raft_consensus.cc:2454] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f60f689a5c74e05ad2d2ca43fcea86d in term 1.
I20260806 16:04:28.692466 25062 leader_election.cc:302] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f60f689a5c74e05ad2d2ca43fcea86d, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:28.692673 25702 raft_consensus.cc:2781] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:28.692953 25702 raft_consensus.cc:686] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 LEADER]: Becoming Leader. State: Replica: 4f60f689a5c74e05ad2d2ca43fcea86d, State: Running, Role: LEADER
I20260806 16:04:28.693344 25702 consensus_queue.cc:237] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } }
I20260806 16:04:28.697311 24919 catalog_manager.cc:5495] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1). New cstate: current_term: 1 leader_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } health_report { overall_health: HEALTHY } } }
I20260806 16:04:28.717061 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:28.713547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.13.148.1:34033)
0806 16:04:28.713699 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:28.713703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:28.713717 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:28.713845 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:28.713848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:28.713904 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:28.713996 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:28.714244 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.714833 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.715687 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:28.715930 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:28.716040 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:28.716277 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:28.716709 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:28.716805 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:28.716879 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 16:04:28.716934 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:04:28.723381 25375 raft_consensus.cc:1261] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Refusing update from remote peer 4f60f689a5c74e05ad2d2ca43fcea86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:28.723383 25577 raft_consensus.cc:1261] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Refusing update from remote peer 4f60f689a5c74e05ad2d2ca43fcea86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:28.724141 25708 consensus_queue.cc:1035] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:28.724418 25702 consensus_queue.cc:1035] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:28.741576 24920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:28.741777 24920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:28.744264 24920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: Sending DeleteTablet for 3 replicas of tablet e846961e79b842c993ee98c651c564df
I20260806 16:04:28.745250 25353 tablet_service.cc:1514] Processing DeleteTablet for tablet e846961e79b842c993ee98c651c564df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:28 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:50486
I20260806 16:04:28.745216 25553 tablet_service.cc:1514] Processing DeleteTablet for tablet e846961e79b842c993ee98c651c564df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:28 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43800
I20260806 16:04:28.745769 25727 tablet_replica.cc:330] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: stopping tablet replica
I20260806 16:04:28.745800 25130 tablet_service.cc:1514] Processing DeleteTablet for tablet e846961e79b842c993ee98c651c564df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:28 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52746
I20260806 16:04:28.746109 25727 raft_consensus.cc:2229] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:28.746346 25728 tablet_replica.cc:330] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: stopping tablet replica
I20260806 16:04:28.746601 25727 raft_consensus.cc:2258] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:28.746827 25728 raft_consensus.cc:2229] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:28.747131 24920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 16:04:28.747502 25728 raft_consensus.cc:2258] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 16:04:28.747589 24920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:28.748257 25727 ts_tablet_manager.cc:1905] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:28.748507 25725 tablet_replica.cc:330] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: stopping tablet replica
I20260806 16:04:28.748780 25725 raft_consensus.cc:2229] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:28.749048 25725 raft_consensus.cc:2258] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:28.750507 25725 ts_tablet_manager.cc:1905] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:28.751204 25727 ts_tablet_manager.cc:1918] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:28.751295 25727 log.cc:1196] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/e846961e79b842c993ee98c651c564df
I20260806 16:04:28.751605 25727 ts_tablet_manager.cc:1939] T e846961e79b842c993ee98c651c564df P 6baca6740ec549559979bcf238b1c292: Deleting consensus metadata
I20260806 16:04:28.752998 24898 catalog_manager.cc:4841] TS 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383): tablet e846961e79b842c993ee98c651c564df (table test-table [id=2d4fcc86f43a4b02b6454dd53a90027c]) successfully deleted
I20260806 16:04:28.754891 25353 tablet_service.cc:1467] Processing CreateTablet for tablet 1329cb81e809464da65e42b45875cf71 (DEFAULT_TABLE table=test-table [id=d7650795078b4e30b55ea05ebe1e1646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:28.754943 25130 tablet_service.cc:1467] Processing CreateTablet for tablet 1329cb81e809464da65e42b45875cf71 (DEFAULT_TABLE table=test-table [id=d7650795078b4e30b55ea05ebe1e1646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:28.755179 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1329cb81e809464da65e42b45875cf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:28.755179 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1329cb81e809464da65e42b45875cf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:28.755185 25553 tablet_service.cc:1467] Processing CreateTablet for tablet 1329cb81e809464da65e42b45875cf71 (DEFAULT_TABLE table=test-table [id=d7650795078b4e30b55ea05ebe1e1646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:28.755450 25553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1329cb81e809464da65e42b45875cf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:28.757968 25688 tablet_bootstrap.cc:492] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: Bootstrap starting.
I20260806 16:04:28.757998 25692 tablet_bootstrap.cc:492] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: Bootstrap starting.
I20260806 16:04:28.758160 25689 tablet_bootstrap.cc:492] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: Bootstrap starting.
I20260806 16:04:28.753932 25725 ts_tablet_manager.cc:1918] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:28.758287 25725 log.cc:1196] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/e846961e79b842c993ee98c651c564df
I20260806 16:04:28.758672 25725 ts_tablet_manager.cc:1939] T e846961e79b842c993ee98c651c564df P 46e9679496e3430590d9cba787e88ccb: Deleting consensus metadata
I20260806 16:04:28.759115 25688 tablet_bootstrap.cc:654] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:28.759138 25692 tablet_bootstrap.cc:654] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:28.759171 25689 tablet_bootstrap.cc:654] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:28.760061 25728 ts_tablet_manager.cc:1905] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:28.760859 24896 catalog_manager.cc:4841] TS 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783): tablet e846961e79b842c993ee98c651c564df (table test-table [id=2d4fcc86f43a4b02b6454dd53a90027c]) successfully deleted
I20260806 16:04:28.761446 25692 tablet_bootstrap.cc:492] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: No bootstrap required, opened a new log
I20260806 16:04:28.761552 25692 ts_tablet_manager.cc:1397] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260806 16:04:28.762038 25692 raft_consensus.cc:348] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:28.762164 25692 raft_consensus.cc:374] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:28.762209 25692 raft_consensus.cc:729] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f60f689a5c74e05ad2d2ca43fcea86d, State: Initialized, Role: FOLLOWER
I20260806 16:04:28.762327 25692 consensus_queue.cc:260] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:28.762681 25692 ts_tablet_manager.cc:1428] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 16:04:28.762755 25728 ts_tablet_manager.cc:1918] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:28.762851 25728 log.cc:1196] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/e846961e79b842c993ee98c651c564df
I20260806 16:04:28.763213 25728 ts_tablet_manager.cc:1939] T e846961e79b842c993ee98c651c564df P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting consensus metadata
I20260806 16:04:28.764158 24896 catalog_manager.cc:4841] TS 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033): tablet e846961e79b842c993ee98c651c564df (table test-table [id=2d4fcc86f43a4b02b6454dd53a90027c]) successfully deleted
I20260806 16:04:28.764973 25689 tablet_bootstrap.cc:492] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: No bootstrap required, opened a new log
I20260806 16:04:28.765064 25689 ts_tablet_manager.cc:1397] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 16:04:28.765204 25688 tablet_bootstrap.cc:492] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: No bootstrap required, opened a new log
I20260806 16:04:28.765282 25688 ts_tablet_manager.cc:1397] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 16:04:28.765538 25689 raft_consensus.cc:348] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:28.765642 25689 raft_consensus.cc:374] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:28.765676 25689 raft_consensus.cc:729] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9679496e3430590d9cba787e88ccb, State: Initialized, Role: FOLLOWER
I20260806 16:04:28.765722 25688 raft_consensus.cc:348] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:28.765821 25688 raft_consensus.cc:374] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:28.765853 25688 raft_consensus.cc:729] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6baca6740ec549559979bcf238b1c292, State: Initialized, Role: FOLLOWER
I20260806 16:04:28.765802 25689 consensus_queue.cc:260] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:28.765971 25688 consensus_queue.cc:260] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:28.766242 25688 ts_tablet_manager.cc:1428] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:04:28.767216 25689 ts_tablet_manager.cc:1428] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260806 16:04:28.779587 25235 tablet.cc:2369] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:04:28.786418 25452 tablet.cc:2369] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:04:29.016885 25654 tablet.cc:2369] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:04:29.046226 25708 raft_consensus.cc:482] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:29.046358 25708 raft_consensus.cc:504] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.046949 25708 leader_election.cc:288] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383), 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:29.047513 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1329cb81e809464da65e42b45875cf71" candidate_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6baca6740ec549559979bcf238b1c292" is_pre_election: true
I20260806 16:04:29.047554 25577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1329cb81e809464da65e42b45875cf71" candidate_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9679496e3430590d9cba787e88ccb" is_pre_election: true
I20260806 16:04:29.047684 25375 raft_consensus.cc:2454] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f60f689a5c74e05ad2d2ca43fcea86d in term 0.
I20260806 16:04:29.047684 25577 raft_consensus.cc:2454] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f60f689a5c74e05ad2d2ca43fcea86d in term 0.
I20260806 16:04:29.048017 25060 leader_election.cc:302] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e9679496e3430590d9cba787e88ccb, 4f60f689a5c74e05ad2d2ca43fcea86d; no voters:
I20260806 16:04:29.048192 25708 raft_consensus.cc:2781] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:29.048285 25708 raft_consensus.cc:482] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:29.048341 25708 raft_consensus.cc:3037] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.049489 25708 raft_consensus.cc:504] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.049881 25708 leader_election.cc:288] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [CANDIDATE]: Term 1 election: Requested vote from peers 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383), 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:29.050244 25577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1329cb81e809464da65e42b45875cf71" candidate_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9679496e3430590d9cba787e88ccb"
I20260806 16:04:29.050244 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1329cb81e809464da65e42b45875cf71" candidate_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6baca6740ec549559979bcf238b1c292"
I20260806 16:04:29.050376 25375 raft_consensus.cc:3037] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.050376 25577 raft_consensus.cc:3037] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.051571 25375 raft_consensus.cc:2454] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f60f689a5c74e05ad2d2ca43fcea86d in term 1.
I20260806 16:04:29.051571 25577 raft_consensus.cc:2454] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f60f689a5c74e05ad2d2ca43fcea86d in term 1.
I20260806 16:04:29.051903 25062 leader_election.cc:302] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f60f689a5c74e05ad2d2ca43fcea86d, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:29.052089 25708 raft_consensus.cc:2781] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:29.052232 25708 raft_consensus.cc:686] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 LEADER]: Becoming Leader. State: Replica: 4f60f689a5c74e05ad2d2ca43fcea86d, State: Running, Role: LEADER
I20260806 16:04:29.052399 25708 consensus_queue.cc:237] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.054105 24920 catalog_manager.cc:5495] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1). New cstate: current_term: 1 leader_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } health_report { overall_health: UNKNOWN } } }
Aug 06 16:04:29 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032267, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:29.161876 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.152447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.13.148.62:44803)
0806 16:04:29.152833 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.152841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.152857 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:04:29.152898 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.152901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.152941 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.153028 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:29.153787 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.154387 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.155267 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.155511 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.155681 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.158377 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:29.158380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:29.158382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:29.158405 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:29.160160 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:29.160644 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:29.160647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:29.160648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:29.160730 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:29.160999 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:29.161001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:29.161002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:29.161204 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:29.161287 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.161581 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.161706 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":312,"thread_start_us":119,"threads_started":1}
I20260806 16:04:29.161891 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.152352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44803 (local address 127.0.0.1:48986)
0806 16:04:29.152666 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:29.152685 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:29.152793 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:29.153108 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:29.153111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:29.153506 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:29.153641 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.153651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.154557 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.154563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.155151 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.155160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.155684 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.155687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.155758 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.156441 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:29.156454 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:29.158196 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:29.160271 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:29.160280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:29.160286 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:29.160552 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:29.160820 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:29.160822 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:29.160823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:29.160914 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:29.161303 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:29.161309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:29.161421 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 16:04:29.161519 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 16:04:29.161640 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
I20260806 16:04:29.166110 24920 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:48986:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 16:04:29.166904 24920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:29.172415 25353 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae2c49a50f44eb84ee548a18efdaa1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0da72f28712d45bcb7ae3e8d97d0c91a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:04:29.172398 25130 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae2c49a50f44eb84ee548a18efdaa1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0da72f28712d45bcb7ae3e8d97d0c91a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:04:29.172614 25553 tablet_service.cc:1467] Processing CreateTablet for tablet f9ae2c49a50f44eb84ee548a18efdaa1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0da72f28712d45bcb7ae3e8d97d0c91a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 16:04:29.172820 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae2c49a50f44eb84ee548a18efdaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.172820 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae2c49a50f44eb84ee548a18efdaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.173022 25553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ae2c49a50f44eb84ee548a18efdaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.175721 25692 tablet_bootstrap.cc:492] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Bootstrap starting.
I20260806 16:04:29.175721 25688 tablet_bootstrap.cc:492] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: Bootstrap starting.
I20260806 16:04:29.175746 25689 tablet_bootstrap.cc:492] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: Bootstrap starting.
I20260806 16:04:29.176574 25689 tablet_bootstrap.cc:654] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.176632 25692 tablet_bootstrap.cc:654] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.176632 25688 tablet_bootstrap.cc:654] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.178167 25692 tablet_bootstrap.cc:492] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: No bootstrap required, opened a new log
I20260806 16:04:29.178190 25688 tablet_bootstrap.cc:492] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: No bootstrap required, opened a new log
I20260806 16:04:29.178277 25692 ts_tablet_manager.cc:1397] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 16:04:29.178287 25688 ts_tablet_manager.cc:1397] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:29.178721 25692 raft_consensus.cc:348] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.178721 25688 raft_consensus.cc:348] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.178839 25688 raft_consensus.cc:374] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.178843 25692 raft_consensus.cc:374] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.178884 25688 raft_consensus.cc:729] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6baca6740ec549559979bcf238b1c292, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.178889 25692 raft_consensus.cc:729] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f60f689a5c74e05ad2d2ca43fcea86d, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.179021 25692 consensus_queue.cc:260] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.179023 25688 consensus_queue.cc:260] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.179308 25688 ts_tablet_manager.cc:1428] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:29.179316 25692 ts_tablet_manager.cc:1428] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 16:04:29.179308 25700 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.179507 25700 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:04:29.179845 25689 tablet_bootstrap.cc:492] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: No bootstrap required, opened a new log
I20260806 16:04:29.179920 25689 ts_tablet_manager.cc:1397] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 16:04:29.180269 25689 raft_consensus.cc:348] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.180358 25689 raft_consensus.cc:374] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.180379 25689 raft_consensus.cc:729] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9679496e3430590d9cba787e88ccb, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.180509 25689 consensus_queue.cc:260] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.180820 25689 ts_tablet_manager.cc:1428] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 16:04:29.180768 25701 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.180904 25701 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:04:29.181031 25708 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.181152 25708 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:04:29.335870 25700 raft_consensus.cc:482] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:29.336035 25700 raft_consensus.cc:504] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.337355 25700 leader_election.cc:288] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033), 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:29.343068 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.338239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:45483 (local address 127.13.148.3:36783)
0806 16:04:29.338561 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.338565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.338580 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:04:29.338635 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.338637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.338681 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.338761 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:29.339029 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.339865 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.341655 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.342364 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.342541 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.342597 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.342817 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.342903 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":120,"threads_started":1}
I20260806 16:04:29.343068 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.337708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:54473 (local address 127.13.148.1:34033)
0806 16:04:29.338000 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.338006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.338026 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:04:29.338267 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.338270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.338327 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.338416 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:29.338873 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.339791 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.341641 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.342353 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.342534 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.342597 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.342821 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.342903 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":64,"threads_started":1}
I20260806 16:04:29.343106 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.338064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36783 (local address 127.13.148.2:45483)
0806 16:04:29.338383 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:29.338399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:29.338523 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:29.338793 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:29.338798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:29.338819 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:29.338913 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.338917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.339965 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.339968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.341490 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.341502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.342517 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.342521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.342642 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.342686 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 16:04:29.342782 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 16:04:29.342918 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":179,"thread_start_us":68,"threads_started":1}
I20260806 16:04:29.343106 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.337583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34033 (local address 127.13.148.2:54473)
0806 16:04:29.337920 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:29.337936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:29.338078 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:29.338569 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:29.338576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:29.338596 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:29.338716 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.338722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.339939 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.339944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.341490 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.341502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.342534 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.342537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.342658 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.342697 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 16:04:29.342771 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 16:04:29.342918 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":221,"thread_start_us":104,"threads_started":1}
I20260806 16:04:29.343608 25577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae2c49a50f44eb84ee548a18efdaa1" candidate_uuid: "6baca6740ec549559979bcf238b1c292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9679496e3430590d9cba787e88ccb" is_pre_election: true
I20260806 16:04:29.343803 25577 raft_consensus.cc:2454] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6baca6740ec549559979bcf238b1c292 in term 0.
I20260806 16:04:29.344151 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae2c49a50f44eb84ee548a18efdaa1" candidate_uuid: "6baca6740ec549559979bcf238b1c292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" is_pre_election: true
I20260806 16:04:29.344219 25276 leader_election.cc:302] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e9679496e3430590d9cba787e88ccb, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:29.344422 25162 raft_consensus.cc:2454] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6baca6740ec549559979bcf238b1c292 in term 0.
I20260806 16:04:29.344527 25700 raft_consensus.cc:2781] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:29.344594 25700 raft_consensus.cc:482] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:29.344647 25700 raft_consensus.cc:3037] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.345678 25700 raft_consensus.cc:504] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.346021 25700 leader_election.cc:288] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [CANDIDATE]: Term 1 election: Requested vote from peers 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033), 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:29.346403 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae2c49a50f44eb84ee548a18efdaa1" candidate_uuid: "6baca6740ec549559979bcf238b1c292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d"
I20260806 16:04:29.346421 25577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ae2c49a50f44eb84ee548a18efdaa1" candidate_uuid: "6baca6740ec549559979bcf238b1c292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9679496e3430590d9cba787e88ccb"
I20260806 16:04:29.346567 25162 raft_consensus.cc:3037] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.346582 25577 raft_consensus.cc:3037] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.347540 25162 raft_consensus.cc:2454] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6baca6740ec549559979bcf238b1c292 in term 1.
I20260806 16:04:29.347541 25577 raft_consensus.cc:2454] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6baca6740ec549559979bcf238b1c292 in term 1.
I20260806 16:04:29.347856 25276 leader_election.cc:302] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f60f689a5c74e05ad2d2ca43fcea86d, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:29.348052 25700 raft_consensus.cc:2781] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:29.348248 25700 raft_consensus.cc:686] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [term 1 LEADER]: Becoming Leader. State: Replica: 6baca6740ec549559979bcf238b1c292, State: Running, Role: LEADER
I20260806 16:04:29.348508 25700 consensus_queue.cc:237] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.349880 25752 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: TxnStatusTablet state changed. Reason: New leader 6baca6740ec549559979bcf238b1c292. Latest consensus state: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.350008 25752 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:04:29.350293 25755 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 16:04:29.350456 25755 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 16:04:29.351373 24918 catalog_manager.cc:5495] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6baca6740ec549559979bcf238b1c292 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } health_report { overall_health: UNKNOWN } } }
I20260806 16:04:29.370888 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.366301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50502 (local address 127.13.148.2:39383)
0806 16:04:29.366622 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.366630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.366646 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:04:29.366729 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.366732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.366806 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.366919 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:29.367185 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.368035 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.368991 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.369389 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.369495 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.369792 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:29.370313 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:29.370426 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.370649 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.370718 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":131,"threads_started":1}
I20260806 16:04:29.370929 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.366186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39383 (local address 127.0.0.1:50502)
0806 16:04:29.366385 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:29.366405 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:29.366560 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:29.366948 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:29.366950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:29.366958 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:04:29.367055 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.367059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.368180 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.368183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.368857 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.368867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.369520 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.369522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.369597 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.369646 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:04:29.370442 (+ 796us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:04:29.370460 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 16:04:29.370625 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 16:04:29.370710 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 16:04:29.374230 25577 raft_consensus.cc:1261] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Refusing update from remote peer 6baca6740ec549559979bcf238b1c292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:29.374550 25162 raft_consensus.cc:1261] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Refusing update from remote peer 6baca6740ec549559979bcf238b1c292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:29.374951 25752 consensus_queue.cc:1035] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:29.375200 25700 consensus_queue.cc:1035] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:29.376708 25701 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: TxnStatusTablet state changed. Reason: New leader 6baca6740ec549559979bcf238b1c292. Latest consensus state: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.376827 25701 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:04:29.378329 25708 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: TxnStatusTablet state changed. Reason: New leader 6baca6740ec549559979bcf238b1c292. Latest consensus state: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.378450 25708 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:04:29.380178 25752 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.380296 25752 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:04:29.380436 25752 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.380556 25752 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 6baca6740ec549559979bcf238b1c292: This TxnStatusTablet replica's current role is: LEADER
I20260806 16:04:29.380584 25701 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.380707 25701 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 46e9679496e3430590d9cba787e88ccb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:04:29.382361 25708 tablet_replica.cc:439] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } } }
I20260806 16:04:29.382512 25708 tablet_replica.cc:442] T f9ae2c49a50f44eb84ee548a18efdaa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 16:04:29.382843 25768 mvcc.cc:205] Tried to move back new op lower bound from 7315588175351287808 to 7315588175254048768. Current Snapshot: MvccSnapshot[applied={T|T < 7315588175351287808}]
I20260806 16:04:29.397502 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.393626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39383 (local address 127.13.148.1:53271)
0806 16:04:29.393977 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:29.394005 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:29.394109 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:29.394346 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:29.394348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:29.394355 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:04:29.394457 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.394507 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.395532 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.395535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.396236 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.396246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.396823 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.396825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.396893 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.396941 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:04:29.397244 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:04:29.397249 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:04:29.397301 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 16:04:29.397370 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":142,"threads_started":1}
I20260806 16:04:29.397506 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.393723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:53271 (local address 127.13.148.2:39383)
0806 16:04:29.393847 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.393851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.394088 (+ 237us) server_negotiation.cc:407] Connection header received
0806 16:04:29.394189 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.394192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.394243 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.394323 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:29.394601 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.395416 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.396355 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.396703 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.396808 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.397072 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:29.397154 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:29.397225 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.397323 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.397374 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:04:29.404436 25577 raft_consensus.cc:1261] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Refusing update from remote peer 4f60f689a5c74e05ad2d2ca43fcea86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:29.404896 25702 consensus_queue.cc:1035] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:29.405382 25375 raft_consensus.cc:1261] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Refusing update from remote peer 4f60f689a5c74e05ad2d2ca43fcea86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:29.405789 25702 consensus_queue.cc:1035] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:29.409488 25765 mvcc.cc:205] Tried to move back new op lower bound from 7315588175477194752 to 7315588174040268800. Current Snapshot: MvccSnapshot[applied={T|T < 7315588175477194752}]
I20260806 16:04:29.410655 25764 mvcc.cc:205] Tried to move back new op lower bound from 7315588175477194752 to 7315588174040268800. Current Snapshot: MvccSnapshot[applied={T|T < 7315588175477194752}]
I20260806 16:04:29.430847 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.426633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34033 (local address 127.13.148.2:56339)
0806 16:04:29.426984 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:29.427002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:29.427118 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:29.427363 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:29.427365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:29.427373 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:04:29.427483 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.427487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.428502 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.428505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.429366 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.429380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.430060 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.430067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.430165 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.430212 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:04:29.430562 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:04:29.430567 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:04:29.430615 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 16:04:29.430682 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
I20260806 16:04:29.430831 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.426689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:56339 (local address 127.13.148.1:34033)
0806 16:04:29.426939 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.426943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.427091 (+ 148us) server_negotiation.cc:407] Connection header received
0806 16:04:29.427207 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.427210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.427263 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.427353 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:29.427582 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.428398 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.429495 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.429850 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.429954 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.430351 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:29.430435 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:29.430531 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.430627 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.430694 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260806 16:04:29.451416 24920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:29.451570 24920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:29.453549 24920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: Sending DeleteTablet for 3 replicas of tablet 1329cb81e809464da65e42b45875cf71
I20260806 16:04:29.454090 25130 tablet_service.cc:1514] Processing DeleteTablet for tablet 1329cb81e809464da65e42b45875cf71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:29 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52746
I20260806 16:04:29.454165 25553 tablet_service.cc:1514] Processing DeleteTablet for tablet 1329cb81e809464da65e42b45875cf71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:29 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43800
I20260806 16:04:29.454233 25353 tablet_service.cc:1514] Processing DeleteTablet for tablet 1329cb81e809464da65e42b45875cf71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:29 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:50486
I20260806 16:04:29.454691 25791 tablet_replica.cc:330] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: stopping tablet replica
I20260806 16:04:29.454854 25791 raft_consensus.cc:2229] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:29.455003 25791 raft_consensus.cc:2258] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:29.455221 25788 tablet_replica.cc:330] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: stopping tablet replica
I20260806 16:04:29.455354 25788 raft_consensus.cc:2229] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:29.455528 25788 raft_consensus.cc:2258] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:29.456204 25788 ts_tablet_manager.cc:1905] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:29.458058 25791 ts_tablet_manager.cc:1905] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:29.458520 25792 tablet_replica.cc:330] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: stopping tablet replica
I20260806 16:04:29.458648 25792 raft_consensus.cc:2229] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:29.458765 25792 raft_consensus.cc:2258] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:29.459378 25792 ts_tablet_manager.cc:1905] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:29.459412 25788 ts_tablet_manager.cc:1918] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:29.459491 25788 log.cc:1196] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/1329cb81e809464da65e42b45875cf71
I20260806 16:04:29.459815 25788 ts_tablet_manager.cc:1939] T 1329cb81e809464da65e42b45875cf71 P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting consensus metadata
I20260806 16:04:29.460533 24896 catalog_manager.cc:4841] TS 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033): tablet 1329cb81e809464da65e42b45875cf71 (table test-table [id=d7650795078b4e30b55ea05ebe1e1646]) successfully deleted
I20260806 16:04:29.461520 25791 ts_tablet_manager.cc:1918] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:29.461634 25791 log.cc:1196] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/1329cb81e809464da65e42b45875cf71
I20260806 16:04:29.461882 25791 ts_tablet_manager.cc:1939] T 1329cb81e809464da65e42b45875cf71 P 46e9679496e3430590d9cba787e88ccb: Deleting consensus metadata
I20260806 16:04:29.462114 25792 ts_tablet_manager.cc:1918] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:29.462198 25792 log.cc:1196] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/1329cb81e809464da65e42b45875cf71
I20260806 16:04:29.462507 25792 ts_tablet_manager.cc:1939] T 1329cb81e809464da65e42b45875cf71 P 6baca6740ec549559979bcf238b1c292: Deleting consensus metadata
I20260806 16:04:29.462651 24896 catalog_manager.cc:4841] TS 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783): tablet 1329cb81e809464da65e42b45875cf71 (table test-table [id=d7650795078b4e30b55ea05ebe1e1646]) successfully deleted
I20260806 16:04:29.463165 24898 catalog_manager.cc:4841] TS 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383): tablet 1329cb81e809464da65e42b45875cf71 (table test-table [id=d7650795078b4e30b55ea05ebe1e1646]) successfully deleted
Aug 06 16:04:29 dist-test-slave-7kzw krb5kdc[24845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 16:04:29 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:29 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:04:29.492571 25743 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:29.485330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.13.148.62:44803)
0806 16:04:29.485500 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.485505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.485851 (+ 346us) server_negotiation.cc:407] Connection header received
0806 16:04:29.486032 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.486036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.486099 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.486187 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:29.487036 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.487598 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.488442 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.488693 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.488829 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.492429 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48988: BlockingRecv error: recv got EOF from 127.0.0.1:48988 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 16:04:29 dist-test-slave-7kzw krb5kdc[24845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260806 16:04:29.503868 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.494161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.13.148.62:44803)
0806 16:04:29.494276 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.494280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.494433 (+ 153us) server_negotiation.cc:407] Connection header received
0806 16:04:29.494577 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.494580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.494632 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.494722 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:29.495279 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.496068 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.496964 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.497290 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.497408 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.500178 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:29.500182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:29.500185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:29.500213 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:29.501948 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:29.502416 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:29.502420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:29.502422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:29.502566 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:29.502912 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:29.502916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:29.502918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:29.503157 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:29.503297 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.503554 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.503699 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:04:29.506786 24920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:29.507294 24920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:29.512393 25130 tablet_service.cc:1467] Processing CreateTablet for tablet 452d52ea8f2d47c5a9afc16904215aa1 (DEFAULT_TABLE table=test-table [id=157bb6bac8ff44e2b55701f3e635618f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:29.512605 25353 tablet_service.cc:1467] Processing CreateTablet for tablet 452d52ea8f2d47c5a9afc16904215aa1 (DEFAULT_TABLE table=test-table [id=157bb6bac8ff44e2b55701f3e635618f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:29.512665 25553 tablet_service.cc:1467] Processing CreateTablet for tablet 452d52ea8f2d47c5a9afc16904215aa1 (DEFAULT_TABLE table=test-table [id=157bb6bac8ff44e2b55701f3e635618f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:29.512766 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452d52ea8f2d47c5a9afc16904215aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.512902 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452d52ea8f2d47c5a9afc16904215aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.512936 25553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 452d52ea8f2d47c5a9afc16904215aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.515324 25692 tablet_bootstrap.cc:492] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Bootstrap starting.
I20260806 16:04:29.515422 25689 tablet_bootstrap.cc:492] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: Bootstrap starting.
I20260806 16:04:29.515517 25688 tablet_bootstrap.cc:492] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: Bootstrap starting.
I20260806 16:04:29.516261 25689 tablet_bootstrap.cc:654] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.516258 25692 tablet_bootstrap.cc:654] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.516429 25688 tablet_bootstrap.cc:654] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.517812 25689 tablet_bootstrap.cc:492] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: No bootstrap required, opened a new log
I20260806 16:04:29.517894 25689 ts_tablet_manager.cc:1397] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:29.518151 25688 tablet_bootstrap.cc:492] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: No bootstrap required, opened a new log
I20260806 16:04:29.518222 25688 ts_tablet_manager.cc:1397] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:29.518308 25689 raft_consensus.cc:348] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.518414 25689 raft_consensus.cc:374] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.518445 25689 raft_consensus.cc:729] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9679496e3430590d9cba787e88ccb, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.518644 25689 consensus_queue.cc:260] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.518935 25689 ts_tablet_manager.cc:1428] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:29.519013 25688 raft_consensus.cc:348] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.519120 25688 raft_consensus.cc:374] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.519150 25688 raft_consensus.cc:729] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6baca6740ec549559979bcf238b1c292, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.519279 25688 consensus_queue.cc:260] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.519516 25688 ts_tablet_manager.cc:1428] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:29.529975 25692 tablet_bootstrap.cc:492] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: No bootstrap required, opened a new log
I20260806 16:04:29.530092 25692 ts_tablet_manager.cc:1397] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20260806 16:04:29.530531 25692 raft_consensus.cc:348] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.530653 25692 raft_consensus.cc:374] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.530684 25692 raft_consensus.cc:729] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f60f689a5c74e05ad2d2ca43fcea86d, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.530817 25692 consensus_queue.cc:260] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.531090 25692 ts_tablet_manager.cc:1428] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:29.631361 25701 raft_consensus.cc:482] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:29.631537 25701 raft_consensus.cc:504] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.632800 25701 leader_election.cc:288] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033), 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383)
I20260806 16:04:29.636732 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.633164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:56633 (local address 127.13.148.1:34033)
0806 16:04:29.633403 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.633410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.633443 (+ 33us) server_negotiation.cc:407] Connection header received
0806 16:04:29.633654 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.633657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.633698 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.633784 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:29.634045 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.634619 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.635740 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.636150 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.636281 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.636336 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.636513 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.636594 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260806 16:04:29.636763 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.633231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:60849 (local address 127.13.148.2:39383)
0806 16:04:29.633367 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.633374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.633587 (+ 213us) server_negotiation.cc:407] Connection header received
0806 16:04:29.633748 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.633750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.633800 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.633884 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:29.634150 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.634676 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.635780 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.636184 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.636315 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.636367 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.636560 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.636642 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:04:29.636759 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.633066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34033 (local address 127.13.148.3:56633)
0806 16:04:29.633363 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:29.633378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:29.633484 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:29.633799 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:29.633802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:29.633818 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:29.633928 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.633933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.634755 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.634758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.635616 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.635626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.636277 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.636280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.636365 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.636406 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 16:04:29.636496 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:04:29.636570 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":5,"thread_start_us":116,"threads_started":1}
I20260806 16:04:29.636803 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.633088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39383 (local address 127.13.148.3:60849)
0806 16:04:29.633488 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:29.633499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:29.633622 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:29.633901 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:29.633904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:29.633921 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:29.634038 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.634044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.634791 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.634797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.635646 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:29.635662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.636308 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.636312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:29.636416 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.636439 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:04:29.636530 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 16:04:29.636600 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":22,"thread_start_us":104,"threads_started":1}
I20260806 16:04:29.637429 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452d52ea8f2d47c5a9afc16904215aa1" candidate_uuid: "46e9679496e3430590d9cba787e88ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" is_pre_election: true
I20260806 16:04:29.637465 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452d52ea8f2d47c5a9afc16904215aa1" candidate_uuid: "46e9679496e3430590d9cba787e88ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6baca6740ec549559979bcf238b1c292" is_pre_election: true
I20260806 16:04:29.637598 25162 raft_consensus.cc:2454] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e9679496e3430590d9cba787e88ccb in term 0.
I20260806 16:04:29.637614 25375 raft_consensus.cc:2454] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e9679496e3430590d9cba787e88ccb in term 0.
I20260806 16:04:29.637981 25483 leader_election.cc:302] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 46e9679496e3430590d9cba787e88ccb, 4f60f689a5c74e05ad2d2ca43fcea86d, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:29.638267 25701 raft_consensus.cc:2781] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:29.638334 25701 raft_consensus.cc:482] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:29.638361 25701 raft_consensus.cc:3037] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.639424 25701 raft_consensus.cc:504] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.639794 25701 leader_election.cc:288] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [CANDIDATE]: Term 1 election: Requested vote from peers 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033), 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383)
I20260806 16:04:29.640133 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452d52ea8f2d47c5a9afc16904215aa1" candidate_uuid: "46e9679496e3430590d9cba787e88ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d"
I20260806 16:04:29.640161 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "452d52ea8f2d47c5a9afc16904215aa1" candidate_uuid: "46e9679496e3430590d9cba787e88ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6baca6740ec549559979bcf238b1c292"
I20260806 16:04:29.640268 25162 raft_consensus.cc:3037] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.640338 25375 raft_consensus.cc:3037] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:29.641229 25375 raft_consensus.cc:2454] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e9679496e3430590d9cba787e88ccb in term 1.
I20260806 16:04:29.641381 25162 raft_consensus.cc:2454] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e9679496e3430590d9cba787e88ccb in term 1.
I20260806 16:04:29.641609 25484 leader_election.cc:302] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e9679496e3430590d9cba787e88ccb, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:29.641790 25701 raft_consensus.cc:2781] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:29.642028 25701 raft_consensus.cc:686] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 1 LEADER]: Becoming Leader. State: Replica: 46e9679496e3430590d9cba787e88ccb, State: Running, Role: LEADER
I20260806 16:04:29.642572 25701 consensus_queue.cc:237] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.644883 24919 catalog_manager.cc:5495] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb reported cstate change: term changed from 0 to 1, leader changed from <none> to 46e9679496e3430590d9cba787e88ccb (127.13.148.3). New cstate: current_term: 1 leader_uuid: "46e9679496e3430590d9cba787e88ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } health_report { overall_health: HEALTHY } } }
I20260806 16:04:29.679581 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:29.676030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.13.148.3:36783)
0806 16:04:29.676171 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:29.676176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:29.676188 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:04:29.676266 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:29.676268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:29.676324 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:29.676442 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:29.676703 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.677208 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.678076 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:29.678291 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:29.678401 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:29.678713 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:29.679177 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:29.679303 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:29.679375 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 16:04:29.679432 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:04:29.682703 25375 raft_consensus.cc:1261] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Refusing update from remote peer 46e9679496e3430590d9cba787e88ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:29.682744 25162 raft_consensus.cc:1261] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Refusing update from remote peer 46e9679496e3430590d9cba787e88ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:29.683285 25817 consensus_queue.cc:1035] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:29.683475 25701 consensus_queue.cc:1035] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:29.695314 24920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:29.695489 24920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:29.697661 24920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: Sending DeleteTablet for 3 replicas of tablet 452d52ea8f2d47c5a9afc16904215aa1
I20260806 16:04:29.698239 25130 tablet_service.cc:1514] Processing DeleteTablet for tablet 452d52ea8f2d47c5a9afc16904215aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:29 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52746
I20260806 16:04:29.698375 25353 tablet_service.cc:1514] Processing DeleteTablet for tablet 452d52ea8f2d47c5a9afc16904215aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:29 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:50486
I20260806 16:04:29.698437 25788 tablet_replica.cc:330] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: stopping tablet replica
I20260806 16:04:29.698585 25788 raft_consensus.cc:2229] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:29.698608 25792 tablet_replica.cc:330] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: stopping tablet replica
I20260806 16:04:29.698566 25553 tablet_service.cc:1514] Processing DeleteTablet for tablet 452d52ea8f2d47c5a9afc16904215aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:29 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43800
I20260806 16:04:29.698731 25788 raft_consensus.cc:2258] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:29.698741 25792 raft_consensus.cc:2229] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:29.698866 25792 raft_consensus.cc:2258] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:29.698891 25791 tablet_replica.cc:330] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: stopping tablet replica
I20260806 16:04:29.698990 25791 raft_consensus.cc:2229] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:29.699285 25791 raft_consensus.cc:2258] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:29.699609 25792 ts_tablet_manager.cc:1905] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:29.700277 25791 ts_tablet_manager.cc:1905] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:29.702282 25792 ts_tablet_manager.cc:1918] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:29.702360 25792 log.cc:1196] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/452d52ea8f2d47c5a9afc16904215aa1
I20260806 16:04:29.702644 25792 ts_tablet_manager.cc:1939] T 452d52ea8f2d47c5a9afc16904215aa1 P 6baca6740ec549559979bcf238b1c292: Deleting consensus metadata
I20260806 16:04:29.702864 25791 ts_tablet_manager.cc:1918] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:29.702936 25791 log.cc:1196] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/452d52ea8f2d47c5a9afc16904215aa1
I20260806 16:04:29.703204 25791 ts_tablet_manager.cc:1939] T 452d52ea8f2d47c5a9afc16904215aa1 P 46e9679496e3430590d9cba787e88ccb: Deleting consensus metadata
I20260806 16:04:29.703327 24898 catalog_manager.cc:4841] TS 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383): tablet 452d52ea8f2d47c5a9afc16904215aa1 (table test-table [id=157bb6bac8ff44e2b55701f3e635618f]) successfully deleted
I20260806 16:04:29.703848 24896 catalog_manager.cc:4841] TS 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783): tablet 452d52ea8f2d47c5a9afc16904215aa1 (table test-table [id=157bb6bac8ff44e2b55701f3e635618f]) successfully deleted
I20260806 16:04:29.704288 25788 ts_tablet_manager.cc:1905] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:29.704774 24920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:29.705201 24920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:29.706852 25788 ts_tablet_manager.cc:1918] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:29.706941 25788 log.cc:1196] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/452d52ea8f2d47c5a9afc16904215aa1
I20260806 16:04:29.707247 25788 ts_tablet_manager.cc:1939] T 452d52ea8f2d47c5a9afc16904215aa1 P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting consensus metadata
I20260806 16:04:29.708276 24896 catalog_manager.cc:4841] TS 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033): tablet 452d52ea8f2d47c5a9afc16904215aa1 (table test-table [id=157bb6bac8ff44e2b55701f3e635618f]) successfully deleted
I20260806 16:04:29.710363 25353 tablet_service.cc:1467] Processing CreateTablet for tablet 0567008e3ce141fc9e4ff1c9d19b273f (DEFAULT_TABLE table=test-table [id=e69dd41fa0b24a99ac31924a915bea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:29.710690 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0567008e3ce141fc9e4ff1c9d19b273f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.713076 25688 tablet_bootstrap.cc:492] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: Bootstrap starting.
I20260806 16:04:29.713919 25688 tablet_bootstrap.cc:654] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.715112 25130 tablet_service.cc:1467] Processing CreateTablet for tablet 0567008e3ce141fc9e4ff1c9d19b273f (DEFAULT_TABLE table=test-table [id=e69dd41fa0b24a99ac31924a915bea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:29.715142 25553 tablet_service.cc:1467] Processing CreateTablet for tablet 0567008e3ce141fc9e4ff1c9d19b273f (DEFAULT_TABLE table=test-table [id=e69dd41fa0b24a99ac31924a915bea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:29.715363 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0567008e3ce141fc9e4ff1c9d19b273f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.715443 25553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0567008e3ce141fc9e4ff1c9d19b273f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:29.717458 25689 tablet_bootstrap.cc:492] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: Bootstrap starting.
I20260806 16:04:29.717535 25692 tablet_bootstrap.cc:492] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: Bootstrap starting.
I20260806 16:04:29.718281 25692 tablet_bootstrap.cc:654] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.718422 25689 tablet_bootstrap.cc:654] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:29.719367 25688 tablet_bootstrap.cc:492] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: No bootstrap required, opened a new log
I20260806 16:04:29.719449 25688 ts_tablet_manager.cc:1397] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 16:04:29.719868 25688 raft_consensus.cc:348] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.720011 25688 raft_consensus.cc:374] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.720047 25688 raft_consensus.cc:729] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6baca6740ec549559979bcf238b1c292, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.720165 25688 consensus_queue.cc:260] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.720417 25688 ts_tablet_manager.cc:1428] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:29.720593 25689 tablet_bootstrap.cc:492] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: No bootstrap required, opened a new log
I20260806 16:04:29.720669 25689 ts_tablet_manager.cc:1397] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:04:29.721069 25689 raft_consensus.cc:348] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.721175 25689 raft_consensus.cc:374] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.721206 25689 raft_consensus.cc:729] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e9679496e3430590d9cba787e88ccb, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.721319 25689 consensus_queue.cc:260] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.721570 25689 ts_tablet_manager.cc:1428] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:29.724830 25692 tablet_bootstrap.cc:492] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: No bootstrap required, opened a new log
I20260806 16:04:29.724920 25692 ts_tablet_manager.cc:1397] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 16:04:29.725314 25692 raft_consensus.cc:348] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.725428 25692 raft_consensus.cc:374] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:29.725459 25692 raft_consensus.cc:729] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f60f689a5c74e05ad2d2ca43fcea86d, State: Initialized, Role: FOLLOWER
I20260806 16:04:29.725584 25692 consensus_queue.cc:260] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:29.725849 25692 ts_tablet_manager.cc:1428] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:04:30.005921 25752 raft_consensus.cc:482] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:30.006111 25752 raft_consensus.cc:504] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:30.006695 25752 leader_election.cc:288] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033), 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:30.007148 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0567008e3ce141fc9e4ff1c9d19b273f" candidate_uuid: "6baca6740ec549559979bcf238b1c292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" is_pre_election: true
I20260806 16:04:30.007190 25577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0567008e3ce141fc9e4ff1c9d19b273f" candidate_uuid: "6baca6740ec549559979bcf238b1c292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9679496e3430590d9cba787e88ccb" is_pre_election: true
I20260806 16:04:30.007356 25162 raft_consensus.cc:2454] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6baca6740ec549559979bcf238b1c292 in term 0.
I20260806 16:04:30.007366 25577 raft_consensus.cc:2454] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6baca6740ec549559979bcf238b1c292 in term 0.
I20260806 16:04:30.007794 25276 leader_election.cc:302] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f60f689a5c74e05ad2d2ca43fcea86d, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:30.007992 25752 raft_consensus.cc:2781] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:30.008086 25752 raft_consensus.cc:482] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:30.008143 25752 raft_consensus.cc:3037] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:30.009403 25752 raft_consensus.cc:504] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:30.009747 25752 leader_election.cc:288] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [CANDIDATE]: Term 1 election: Requested vote from peers 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033), 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783)
I20260806 16:04:30.010138 25577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0567008e3ce141fc9e4ff1c9d19b273f" candidate_uuid: "6baca6740ec549559979bcf238b1c292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e9679496e3430590d9cba787e88ccb"
I20260806 16:04:30.010272 25577 raft_consensus.cc:3037] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:30.011246 25577 raft_consensus.cc:2454] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6baca6740ec549559979bcf238b1c292 in term 1.
I20260806 16:04:30.011574 25276 leader_election.cc:302] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e9679496e3430590d9cba787e88ccb, 6baca6740ec549559979bcf238b1c292; no voters:
I20260806 16:04:30.011555 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0567008e3ce141fc9e4ff1c9d19b273f" candidate_uuid: "6baca6740ec549559979bcf238b1c292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d"
I20260806 16:04:30.011724 25162 raft_consensus.cc:3037] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:30.011786 25752 raft_consensus.cc:2781] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:30.011912 25752 raft_consensus.cc:686] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 1 LEADER]: Becoming Leader. State: Replica: 6baca6740ec549559979bcf238b1c292, State: Running, Role: LEADER
I20260806 16:04:30.012293 25752 consensus_queue.cc:237] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } }
I20260806 16:04:30.012816 25162 raft_consensus.cc:2454] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6baca6740ec549559979bcf238b1c292 in term 1.
I20260806 16:04:30.013989 24919 catalog_manager.cc:5495] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6baca6740ec549559979bcf238b1c292 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "6baca6740ec549559979bcf238b1c292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6baca6740ec549559979bcf238b1c292" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 } health_report { overall_health: UNKNOWN } } }
I20260806 16:04:30.087090 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:30.083835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.13.148.2:39383)
0806 16:04:30.083957 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:30.083960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:30.084057 (+ 97us) server_negotiation.cc:407] Connection header received
0806 16:04:30.084238 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:30.084240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:30.084287 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:30.084351 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:30.084618 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:30.085107 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:30.086002 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:30.086218 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:30.086294 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:30.086720 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:30.086785 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:30.086839 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:30.086939 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 16:04:30.086981 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:04:30.096169 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:30.093159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:49585 (local address 127.13.148.2:39383)
0806 16:04:30.093276 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:30.093279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:30.093494 (+ 215us) server_negotiation.cc:407] Connection header received
0806 16:04:30.093597 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:30.093599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:30.093639 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:30.093707 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:30.093902 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:30.094348 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:30.095253 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:30.095464 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:30.095545 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:30.095797 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:30.095853 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:30.095906 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:30.095995 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 16:04:30.096035 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260806 16:04:30.096197 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:30.093073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39383 (local address 127.13.148.2:49585)
0806 16:04:30.093413 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:30.093435 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:30.093524 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:30.093738 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:30.093741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:30.093748 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 16:04:30.093830 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:30.093834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:30.094436 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:30.094438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:30.095129 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:30.095138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:30.095572 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:30.095575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:30.095638 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:30.095696 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 16:04:30.095932 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 16:04:30.095937 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:04:30.095990 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 16:04:30.096058 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":146,"threads_started":1}
I20260806 16:04:30.101776 25577 raft_consensus.cc:1261] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Refusing update from remote peer 6baca6740ec549559979bcf238b1c292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:30.101845 25162 raft_consensus.cc:1261] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Refusing update from remote peer 6baca6740ec549559979bcf238b1c292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:30.102226 25831 consensus_queue.cc:1035] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f60f689a5c74e05ad2d2ca43fcea86d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:30.102387 25752 consensus_queue.cc:1035] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e9679496e3430590d9cba787e88ccb" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:30.137271 24920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:30.137423 24920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:30.139652 24920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3927ff48d6814166b137039c8a48dc38: Sending DeleteTablet for 3 replicas of tablet 0567008e3ce141fc9e4ff1c9d19b273f
I20260806 16:04:30.140261 25353 tablet_service.cc:1514] Processing DeleteTablet for tablet 0567008e3ce141fc9e4ff1c9d19b273f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:30 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:50486
I20260806 16:04:30.140462 25792 tablet_replica.cc:330] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: stopping tablet replica
I20260806 16:04:30.140451 25130 tablet_service.cc:1514] Processing DeleteTablet for tablet 0567008e3ce141fc9e4ff1c9d19b273f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:30 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52746
I20260806 16:04:30.140595 25792 raft_consensus.cc:2229] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:30.140678 25553 tablet_service.cc:1514] Processing DeleteTablet for tablet 0567008e3ce141fc9e4ff1c9d19b273f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:30 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43800
I20260806 16:04:30.140882 25788 tablet_replica.cc:330] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: stopping tablet replica
I20260806 16:04:30.140969 25792 raft_consensus.cc:2258] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:30.141009 25788 raft_consensus.cc:2229] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:30.141130 25788 raft_consensus.cc:2258] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:30.141297 25791 tablet_replica.cc:330] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: stopping tablet replica
I20260806 16:04:30.141417 25791 raft_consensus.cc:2229] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:30.141536 25791 raft_consensus.cc:2258] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:30.141691 25792 ts_tablet_manager.cc:1905] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:30.142314 25788 ts_tablet_manager.cc:1905] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:30.142649 25791 ts_tablet_manager.cc:1905] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:30.144467 25792 ts_tablet_manager.cc:1918] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:30.144538 25792 log.cc:1196] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/0567008e3ce141fc9e4ff1c9d19b273f
I20260806 16:04:30.144526 25791 ts_tablet_manager.cc:1918] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:30.144582 25791 log.cc:1196] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/0567008e3ce141fc9e4ff1c9d19b273f
I20260806 16:04:30.144778 25791 ts_tablet_manager.cc:1939] T 0567008e3ce141fc9e4ff1c9d19b273f P 46e9679496e3430590d9cba787e88ccb: Deleting consensus metadata
I20260806 16:04:30.144830 25792 ts_tablet_manager.cc:1939] T 0567008e3ce141fc9e4ff1c9d19b273f P 6baca6740ec549559979bcf238b1c292: Deleting consensus metadata
I20260806 16:04:30.145217 25788 ts_tablet_manager.cc:1918] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 16:04:30.145287 25788 log.cc:1196] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/0567008e3ce141fc9e4ff1c9d19b273f
I20260806 16:04:30.145550 24896 catalog_manager.cc:4841] TS 46e9679496e3430590d9cba787e88ccb (127.13.148.3:36783): tablet 0567008e3ce141fc9e4ff1c9d19b273f (table test-table [id=e69dd41fa0b24a99ac31924a915bea2e]) successfully deleted
I20260806 16:04:30.145593 24898 catalog_manager.cc:4841] TS 6baca6740ec549559979bcf238b1c292 (127.13.148.2:39383): tablet 0567008e3ce141fc9e4ff1c9d19b273f (table test-table [id=e69dd41fa0b24a99ac31924a915bea2e]) successfully deleted
I20260806 16:04:30.145948 25788 ts_tablet_manager.cc:1939] T 0567008e3ce141fc9e4ff1c9d19b273f P 4f60f689a5c74e05ad2d2ca43fcea86d: Deleting consensus metadata
I20260806 16:04:30.146121 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 25032
W20260806 16:04:30.156498 24896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:34033 (error 108)
I20260806 16:04:30.156625 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 25247
I20260806 16:04:30.162451 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:30.159253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34033 (local address 127.0.0.1:52758)
0806 16:04:30.162282 (+ 3029us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:30.162354 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2952,"thread_start_us":937,"threads_started":1}
W20260806 16:04:30.162644 24896 catalog_manager.cc:4584] TS 4f60f689a5c74e05ad2d2ca43fcea86d (127.13.148.1:34033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0567008e3ce141fc9e4ff1c9d19b273f: Network error: Client connection negotiation failed: client connection to 127.13.148.1:34033: connect: Connection refused (error 111)
I20260806 16:04:30.169713 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 25461
I20260806 16:04:30.185681 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 24866
2026-08-06T16:04:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4058 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:30 dist-test-slave-7kzw krb5kdc[25849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:30 dist-test-slave-7kzw krb5kdc[25849](info): set up 2 sockets
Aug 06 16:04:30 dist-test-slave-7kzw krb5kdc[25849](info): commencing operation
krb5kdc: starting...
W20260806 16:04:31.458148 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:31 dist-test-slave-7kzw krb5kdc[25849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:31.598536 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:43521
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:43521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:31.716701 25864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:31.717031 25864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:31.717132 25864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:31.721247 25864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:31.721345 25864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:31.721382 25864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:31.721441 25864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:31.721494 25864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:31.726930 25864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:43521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:43521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:31.728072 25864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:31.729143 25864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:31.737721 25873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:31.737753 25871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:31.737759 25869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:31.737957 25864 server_base.cc:1034] running on GCE node
I20260806 16:04:31.738703 25864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:31.739828 25864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:31.740999 25864 hybrid_clock.cc:648] HybridClock initialized: now 1786032271740986 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T16:04:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1563 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:31 dist-test-slave-7kzw krb5kdc[25880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:31 dist-test-slave-7kzw krb5kdc[25880](info): set up 2 sockets
Aug 06 16:04:31 dist-test-slave-7kzw krb5kdc[25880](info): commencing operation
krb5kdc: starting...
W20260806 16:04:32.963442 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:33 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:33.101523 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46081
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:33.219316 25897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:33.219626 25897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:33.219713 25897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:33.223749 25897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:33.223826 25897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:33.223865 25897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:33.223922 25897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:33.223961 25897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:33.229388 25897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:33.230566 25897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:33.231657 25897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:33.240634 25908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:33.240682 25906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:33.240703 25904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:33.240856 25897 server_base.cc:1034] running on GCE node
I20260806 16:04:33.241609 25897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:33.242856 25897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:33.244068 25897 hybrid_clock.cc:648] HybridClock initialized: now 1786032273244047 us; error 31 us; skew 500 ppm
Aug 06 16:04:33 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032273, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:33.247195 25897 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:33.248181 25897 webserver.cc:460] Webserver started at http://127.13.148.62:34849/ using document root <none> and password file <none>
I20260806 16:04:33.248821 25897 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:33.248912 25897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:33.249151 25897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:33.251315 25897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "61078b221b1ab62467e1b721480995f2"
server_key_iv: "a66b46d158e8fca5920f69ed3aa5ba7f"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.251899 25897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "61078b221b1ab62467e1b721480995f2"
server_key_iv: "a66b46d158e8fca5920f69ed3aa5ba7f"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.255779 25897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:33.258363 25917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:33.259843 25897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:33.260010 25897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "61078b221b1ab62467e1b721480995f2"
server_key_iv: "a66b46d158e8fca5920f69ed3aa5ba7f"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.260142 25897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:33.284766 25897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:33.288305 25897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:33.288539 25897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:33.296211 25897 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46081
I20260806 16:04:33.296241 25994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46081 every 8 connection(s)
I20260806 16:04:33.297195 25897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:33.299606 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 25897
I20260806 16:04:33.299714 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:33.300017 13904 external_mini_cluster.cc:1428] Setting key 4b2da10831309c0e4dcb9d0b6223bfd8
I20260806 16:04:33.300231 25996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:33.308285 25996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Bootstrap starting.
Aug 06 16:04:33 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:33.311105 25996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:33.312111 25996 log.cc:824] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:33.314678 25996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: No bootstrap required, opened a new log
I20260806 16:04:33.317137 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:33.301488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.13.148.62:46081)
0806 16:04:33.301961 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:33.301981 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:33.302026 (+ 45us) server_negotiation.cc:407] Connection header received
0806 16:04:33.303258 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:33.303285 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:33.303733 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:33.304130 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:33.304836 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.306600 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.307494 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.308240 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.308423 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:33.311526 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:33.311533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:33.311551 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:33.311584 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:33.313741 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:33.314234 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:33.314240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:33.314246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:33.314401 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:33.314886 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:33.314890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:33.314893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:33.315381 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:33.315617 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:33.316047 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 16:04:33.316373 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":123,"threads_started":1}
I20260806 16:04:33.318668 25996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } }
I20260806 16:04:33.318926 25996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:33.319013 25996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3f3dbccabb4cc5a02c3b3a7887cff9, State: Initialized, Role: FOLLOWER
I20260806 16:04:33.319464 25996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } }
I20260806 16:04:33.319629 25996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:33.319708 25996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:33.319820 25996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:33.320791 25996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } }
I20260806 16:04:33.321148 25996 leader_election.cc:302] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e3f3dbccabb4cc5a02c3b3a7887cff9; no voters:
I20260806 16:04:33.321522 25996 leader_election.cc:288] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:33.321704 26004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:33.321995 26004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 1 LEADER]: Becoming Leader. State: Replica: 6e3f3dbccabb4cc5a02c3b3a7887cff9, State: Running, Role: LEADER
I20260806 16:04:33.322335 26004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } }
I20260806 16:04:33.322878 25996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:33.324391 26006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } } }
I20260806 16:04:33.324379 26007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e3f3dbccabb4cc5a02c3b3a7887cff9. Latest consensus state: current_term: 1 leader_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } } }
I20260806 16:04:33.324572 26007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:33.324851 26006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:33.325187 26019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:33.328637 26019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:33.334836 26019 catalog_manager.cc:1347] Generated new cluster ID: 53f34282aa5c46ffb0bfdc12e91ed241
I20260806 16:04:33.334944 26019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:33.358631 26019 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:33.359676 26019 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:33.369254 26019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Generated new TSK 0
I20260806 16:04:33.370112 26019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:04:33.419818 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46081
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:33.553562 26036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:33.553864 26036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:33.553948 26036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:33.558152 26036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:33.558226 26036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:33.558328 26036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:33.563650 26036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:33.564781 26036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:33.565834 26036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:33.576838 26042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:33.576984 26036 server_base.cc:1034] running on GCE node
W20260806 16:04:33.576845 26046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:33.577193 26043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:33.577625 26036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:33.578297 26036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:33.579520 26036 hybrid_clock.cc:648] HybridClock initialized: now 1786032273579500 us; error 32 us; skew 500 ppm
Aug 06 16:04:33 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032273, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:33.582533 26036 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:04:33.583462 26036 webserver.cc:460] Webserver started at http://127.13.148.1:37517/ using document root <none> and password file <none>
I20260806 16:04:33.583995 26036 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:33.584056 26036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:33.584223 26036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:33.586186 26036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "115233f7d362414ca120dd3ab286e7da"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "693d3d7bfd243cdc93c9525a116129d4"
server_key_iv: "a0da6dbee12ce03194f0c74e954ee3a6"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.586786 26036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "115233f7d362414ca120dd3ab286e7da"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "693d3d7bfd243cdc93c9525a116129d4"
server_key_iv: "a0da6dbee12ce03194f0c74e954ee3a6"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.590559 26036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:04:33.593043 26055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:33.594324 26036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:04:33.594461 26036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "115233f7d362414ca120dd3ab286e7da"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "693d3d7bfd243cdc93c9525a116129d4"
server_key_iv: "a0da6dbee12ce03194f0c74e954ee3a6"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.594588 26036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:33.604208 26036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:33.606983 26036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:33.607177 26036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:33.607820 26036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:33.608636 26036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:33.608685 26036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:33.608762 26036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:33.608794 26036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:33.618115 26036 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:41021
I20260806 16:04:33.618186 26211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:41021 every 8 connection(s)
I20260806 16:04:33.619078 26036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:33.628166 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 26036
I20260806 16:04:33.628330 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:33.628654 13904 external_mini_cluster.cc:1428] Setting key 43171751d70e16f6b9e378703b4b03fe
Aug 06 16:04:33 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032273, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:33.633961 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:33.621035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:52679 (local address 127.13.148.62:46081)
0806 16:04:33.621186 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:33.621190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:33.621994 (+ 804us) server_negotiation.cc:407] Connection header received
0806 16:04:33.623526 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:33.623529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:33.623599 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:33.623694 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:33.624950 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.625507 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.626695 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.626931 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.627016 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:33.629840 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:33.629844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:33.629846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:33.629872 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:33.631787 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:33.632339 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:33.632345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:33.632351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:33.632460 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:33.632845 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:33.632849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:33.632851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:33.633045 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:33.633168 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:33.633660 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 16:04:33.633791 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:04:33.635049 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:33.621382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:52679)
0806 16:04:33.621834 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:33.621879 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:33.623258 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:33.623867 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:33.623874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:33.624248 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:33.624804 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:33.624819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.625649 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.625652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:33.626580 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:33.626591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.627102 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.627106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:33.627253 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:33.627913 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:33.627929 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:33.629667 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:33.631915 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:33.631920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:33.631931 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:33.632229 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:33.632571 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:33.632574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:33.632576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:33.632727 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:33.633168 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:33.633176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:33.633412 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 16:04:33.633611 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 16:04:33.633930 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
I20260806 16:04:33.636628 26213 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46081
I20260806 16:04:33.636919 26213 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:33.637452 26213 heartbeater.cc:511] Master 127.13.148.62:46081 requested a full tablet report, sending...
I20260806 16:04:33.639297 25943 ts_manager.cc:194] Registered new tserver with Master: 115233f7d362414ca120dd3ab286e7da (127.13.148.1:41021)
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:04:33.641027 25943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:52679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:04:33.662643 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46081
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:33.779364 26221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:33.779662 26221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:33.779755 26221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:33.783766 26221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:33.783846 26221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:33.783929 26221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:33.789202 26221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:33.790310 26221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:33.791574 26221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:33.802038 26233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:33.802038 26230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:33.802246 26221 server_base.cc:1034] running on GCE node
W20260806 16:04:33.802376 26227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:33.803012 26221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:33.803696 26221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:33.804886 26221 hybrid_clock.cc:648] HybridClock initialized: now 1786032273804864 us; error 35 us; skew 500 ppm
Aug 06 16:04:33 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032273, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:33.808012 26221 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:04:33.808969 26221 webserver.cc:460] Webserver started at http://127.13.148.2:44847/ using document root <none> and password file <none>
I20260806 16:04:33.809557 26221 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:33.809669 26221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:33.809911 26221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:33.812065 26221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "e6a822e185fc44b2afce1afca9e08605"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "35e147f9a881fa8ff851a19dcb3ea39d"
server_key_iv: "a59b9a99e70badaa3a0b1458c63ea045"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.812655 26221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "e6a822e185fc44b2afce1afca9e08605"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "35e147f9a881fa8ff851a19dcb3ea39d"
server_key_iv: "a59b9a99e70badaa3a0b1458c63ea045"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.816427 26221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260806 16:04:33.818974 26248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:33.820202 26221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:04:33.820338 26221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "e6a822e185fc44b2afce1afca9e08605"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "35e147f9a881fa8ff851a19dcb3ea39d"
server_key_iv: "a59b9a99e70badaa3a0b1458c63ea045"
server_key_version: "encryptionkey@0"
I20260806 16:04:33.820449 26221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:33.836964 26221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:33.840044 26221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:33.840246 26221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:33.840847 26221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:33.841696 26221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:33.841768 26221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:33.841835 26221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:33.841882 26221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:33.851128 26221 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:42873
I20260806 16:04:33.851164 26407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:42873 every 8 connection(s)
I20260806 16:04:33.852018 26221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:33.861451 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 26221
I20260806 16:04:33.861594 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:33.861874 13904 external_mini_cluster.cc:1428] Setting key 1fcb6dd382abd0a5d27b8bb7e11489b7
Aug 06 16:04:33 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032273, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:33.866742 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:33.853801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:46889 (local address 127.13.148.62:46081)
0806 16:04:33.853947 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:33.853951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:33.854766 (+ 815us) server_negotiation.cc:407] Connection header received
0806 16:04:33.856091 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:33.856094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:33.856156 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:33.856271 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:33.857419 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.858204 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.859553 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.859804 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.859923 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:33.862787 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:33.862791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:33.862793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:33.862819 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:33.864518 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:33.865077 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:33.865082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:33.865084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:33.865176 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:33.865536 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:33.865540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:33.865541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:33.865722 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:33.865849 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:33.866370 (+ 521us) server_negotiation.cc:299] Negotiation successful
0806 16:04:33.866577 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:04:33.867527 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:33.854106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:46889)
0806 16:04:33.854608 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:33.854651 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:33.855876 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:33.856445 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:33.856452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:33.856804 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:33.857268 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:33.857281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.858342 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.858344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:33.859420 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:33.859431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:33.859965 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:33.859969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:33.860142 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:33.860762 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:33.860778 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:33.862598 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:33.864636 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:33.864643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:33.864658 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:33.864954 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:33.865268 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:33.865272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:33.865274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:33.865423 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:33.865848 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:33.865856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:33.866117 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 16:04:33.866308 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 16:04:33.866669 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260806 16:04:33.869086 26408 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46081
I20260806 16:04:33.869341 26408 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:33.869897 26408 heartbeater.cc:511] Master 127.13.148.62:46081 requested a full tablet report, sending...
I20260806 16:04:33.871142 25943 ts_manager.cc:194] Registered new tserver with Master: e6a822e185fc44b2afce1afca9e08605 (127.13.148.2:42873)
I20260806 16:04:33.871776 25943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:46889
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:04:33.894814 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46081
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:34.011898 26416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:34.012176 26416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:34.012219 26416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:34.016223 26416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:34.016278 26416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:34.016367 26416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:34.021761 26416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:34.022893 26416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:34.024000 26416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:34.034268 26427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:34.034288 26424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:34.034262 26425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:34.034592 26416 server_base.cc:1034] running on GCE node
I20260806 16:04:34.035091 26416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:34.035811 26416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:34.036980 26416 hybrid_clock.cc:648] HybridClock initialized: now 1786032274036950 us; error 42 us; skew 500 ppm
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032274, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:34.040220 26416 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:04:34.041163 26416 webserver.cc:460] Webserver started at http://127.13.148.3:38515/ using document root <none> and password file <none>
I20260806 16:04:34.041797 26416 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:34.041905 26416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:34.042148 26416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:34.044440 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "013a651f991d470db918b1519611615d"
format_stamp: "Formatted at 2026-08-06 16:04:34 on dist-test-slave-7kzw"
server_key: "49015dda87945cba134bc8ecde8f5ade"
server_key_iv: "155575f0fe694610d2a1110b8e0d5078"
server_key_version: "encryptionkey@0"
I20260806 16:04:34.045070 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "013a651f991d470db918b1519611615d"
format_stamp: "Formatted at 2026-08-06 16:04:34 on dist-test-slave-7kzw"
server_key: "49015dda87945cba134bc8ecde8f5ade"
server_key_iv: "155575f0fe694610d2a1110b8e0d5078"
server_key_version: "encryptionkey@0"
I20260806 16:04:34.049180 26416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 16:04:34.051795 26435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.053125 26416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:04:34.053267 26416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "013a651f991d470db918b1519611615d"
format_stamp: "Formatted at 2026-08-06 16:04:34 on dist-test-slave-7kzw"
server_key: "49015dda87945cba134bc8ecde8f5ade"
server_key_iv: "155575f0fe694610d2a1110b8e0d5078"
server_key_version: "encryptionkey@0"
I20260806 16:04:34.053383 26416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:34.072108 26416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:34.075947 26416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:34.076167 26416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:34.076788 26416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:34.077641 26416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:34.077714 26416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.077783 26416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:34.077831 26416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.087262 26416 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:42035
I20260806 16:04:34.087311 26589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:42035 every 8 connection(s)
I20260806 16:04:34.088174 26416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:34.092958 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 26416
I20260806 16:04:34.093106 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:34.093400 13904 external_mini_cluster.cc:1428] Setting key 632b77f0adbe76903961e2c6f4a570f4
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032274, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:34.102703 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:34.090043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:45835 (local address 127.13.148.62:46081)
0806 16:04:34.090252 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:34.090256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:34.091062 (+ 806us) server_negotiation.cc:407] Connection header received
0806 16:04:34.092406 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:34.092409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:34.092472 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:34.092630 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:34.093925 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.094613 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.096058 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.096291 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.096403 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.098949 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:34.098952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:34.098954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:34.098978 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:34.100538 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:34.101093 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:34.101096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:34.101098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:34.101186 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:34.101525 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:34.101528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:34.101529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:34.101758 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:34.101840 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:34.102359 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 16:04:34.102531 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 16:04:34.103502 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:34.090379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:45835)
0806 16:04:34.090900 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:34.090943 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:34.092188 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:34.092814 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:34.092822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:34.093267 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:34.093777 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.093790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.094735 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.094739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.095933 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.095942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.096440 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.096443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.096614 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.097169 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:34.097185 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:34.098793 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:34.100705 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:34.100710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:34.100723 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:34.100991 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:34.101283 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:34.101286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:34.101287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:34.101428 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:34.101876 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:34.101883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:34.102112 (+ 229us) client_negotiation.cc:769] Sending connection context
0806 16:04:34.102278 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 16:04:34.102637 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":122,"threads_started":1}
I20260806 16:04:34.104699 26591 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46081
I20260806 16:04:34.104985 26591 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:34.105494 26591 heartbeater.cc:511] Master 127.13.148.62:46081 requested a full tablet report, sending...
I20260806 16:04:34.106629 25943 ts_manager.cc:194] Registered new tserver with Master: 013a651f991d470db918b1519611615d (127.13.148.3:42035)
I20260806 16:04:34.107317 25943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:45835
I20260806 16:04:34.107703 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
WARNING: no policy specified for oryx/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
WARNING: no policy specified for oryx/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
I20260806 16:04:34.189713 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 26036
I20260806 16:04:34.220901 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 26221
I20260806 16:04:34.235419 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 26416
I20260806 16:04:34.260489 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 25897
I20260806 16:04:34.271749 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46081
--webserver_interface=127.13.148.62
--webserver_port=34849
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:34.383882 26603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:34.384192 26603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:34.384276 26603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:34.388273 26603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:34.388362 26603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:34.388401 26603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:34.388460 26603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:34.388500 26603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:34.393649 26603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=34849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:34.394713 26603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:34.395717 26603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:34.404320 26610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:34.404404 26611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:34.404599 26603 server_base.cc:1034] running on GCE node
W20260806 16:04:34.404354 26614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:34.405129 26603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:34.406152 26603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:34.407347 26603 hybrid_clock.cc:648] HybridClock initialized: now 1786032274407335 us; error 27 us; skew 500 ppm
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032274, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:34.410081 26603 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:34.411062 26603 webserver.cc:460] Webserver started at http://127.13.148.62:34849/ using document root <none> and password file <none>
I20260806 16:04:34.411631 26603 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:34.411741 26603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:34.415520 26603 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 16:04:34.417665 26624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.418881 26603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:04:34.419032 26603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "61078b221b1ab62467e1b721480995f2"
server_key_iv: "a66b46d158e8fca5920f69ed3aa5ba7f"
server_key_version: "encryptionkey@0"
I20260806 16:04:34.419565 26603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:34.431342 26603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:34.434098 26603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:34.434309 26603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:34.441552 26603 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46081
I20260806 16:04:34.441581 26694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46081 every 8 connection(s)
I20260806 16:04:34.442618 26603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:34.447166 26695 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:04:34.448729 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 26603
I20260806 16:04:34.449074 26695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Bootstrap starting.
I20260806 16:04:34.449255 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:41021
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:46081
--webserver_port=37517
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.1
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:04:34.466543 26695 log.cc:824] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:34.473662 26695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:04:34.474083 26695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Bootstrap complete.
I20260806 16:04:34.478420 26695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } }
I20260806 16:04:34.478922 26695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3f3dbccabb4cc5a02c3b3a7887cff9, State: Initialized, Role: FOLLOWER
I20260806 16:04:34.479411 26695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } }
I20260806 16:04:34.479584 26695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:34.479671 26695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:34.479799 26695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:04:34.481781 26695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } }
I20260806 16:04:34.482132 26695 leader_election.cc:302] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e3f3dbccabb4cc5a02c3b3a7887cff9; no voters:
I20260806 16:04:34.482529 26695 leader_election.cc:288] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 16:04:34.482621 26699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:04:34.482820 26699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [term 2 LEADER]: Becoming Leader. State: Replica: 6e3f3dbccabb4cc5a02c3b3a7887cff9, State: Running, Role: LEADER
I20260806 16:04:34.483126 26699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } }
I20260806 16:04:34.483807 26695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:34.484460 26700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } } }
I20260806 16:04:34.484583 26700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:34.485267 26708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:34.485231 26702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e3f3dbccabb4cc5a02c3b3a7887cff9. Latest consensus state: current_term: 2 leader_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e3f3dbccabb4cc5a02c3b3a7887cff9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46081 } } }
I20260806 16:04:34.485859 26702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:34.488619 26708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:34.489277 26708 catalog_manager.cc:1259] Loaded cluster ID: 53f34282aa5c46ffb0bfdc12e91ed241
I20260806 16:04:34.489336 26708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:34.491691 26708 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:34.492362 26708 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e3f3dbccabb4cc5a02c3b3a7887cff9: Loaded TSK: 0
I20260806 16:04:34.493521 26708 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:04:34.568210 26697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:34.568447 26697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:34.568490 26697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:34.573205 26697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:34.573289 26697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:34.573381 26697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:34.579051 26697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:41021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=37517
--tserver_master_addrs=127.13.148.62:46081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:34.580049 26697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:34.581049 26697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:34.591012 26733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:34.590992 26732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:34.591171 26697 server_base.cc:1034] running on GCE node
W20260806 16:04:34.591012 26736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:34.591830 26697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:34.592495 26697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:34.593722 26697 hybrid_clock.cc:648] HybridClock initialized: now 1786032274593690 us; error 53 us; skew 500 ppm
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032274, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:34.597148 26697 init.cc:377] Logged in from keytab as oryx/127.13.148.1@KRBTEST.COM (short username oryx)
I20260806 16:04:34.598264 26697 webserver.cc:460] Webserver started at http://127.13.148.1:37517/ using document root <none> and password file <none>
I20260806 16:04:34.599006 26697 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:34.599135 26697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:34.604478 26697 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:04:34.607204 26749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.608779 26697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:34.608928 26697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "115233f7d362414ca120dd3ab286e7da"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "693d3d7bfd243cdc93c9525a116129d4"
server_key_iv: "a0da6dbee12ce03194f0c74e954ee3a6"
server_key_version: "encryptionkey@0"
I20260806 16:04:34.609663 26697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:34.636170 26697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:34.639860 26697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:34.640087 26697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:34.640810 26697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:34.641753 26697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:34.641805 26697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.641846 26697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:34.641904 26697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.651410 26697 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:41021
I20260806 16:04:34.651444 26903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:41021 every 8 connection(s)
I20260806 16:04:34.652411 26697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:34.657905 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 26697
I20260806 16:04:34.658362 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:42873
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:46081
--webserver_port=44847
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.2
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:04:34.666281 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:34.654572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:43405 (local address 127.13.148.62:46081)
0806 16:04:34.655051 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:34.655067 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:34.655282 (+ 215us) server_negotiation.cc:407] Connection header received
0806 16:04:34.656888 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:34.656901 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:34.657194 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:34.657473 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:34.658751 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.659707 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.660988 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.661534 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.661718 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.665594 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:43405: BlockingRecv error: recv got EOF from 127.13.148.1:43405 (error 108)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":122,"threads_started":1}
I20260806 16:04:34.666409 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:34.654621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:43405)
0806 16:04:34.655113 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:34.655166 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:34.656409 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:34.657573 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:34.657579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:34.657940 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:34.658514 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.658533 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.659851 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.659854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.660802 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.660812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.661713 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.661718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.661932 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.662589 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:34.662606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:34.665395 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":166,"threads_started":1}
W20260806 16:04:34.667109 26905 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:46081 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.62:46081: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:34.672706 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:34.667606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:50463)
0806 16:04:34.667729 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:34.667743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:34.667819 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:34.668056 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:34.668059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:34.668227 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:34.668311 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.668316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.669113 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.669119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.669648 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.669656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.670123 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.670126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.670219 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.670783 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:34.670796 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:34.672550 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:04:34.672794 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:34.667715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:50463 (local address 127.13.148.62:46081)
0806 16:04:34.667830 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:34.667834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:34.667860 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:04:34.667898 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:34.667901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:34.667948 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:34.668033 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:34.668425 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.669000 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.669779 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.670007 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.670112 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.672675 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:50463: BlockingRecv error: recv got EOF from 127.13.148.1:50463 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:34.678572 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:34.673252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:39441)
0806 16:04:34.673376 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:34.673391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:34.673469 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:34.673694 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:34.673697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:34.673858 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:34.673946 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.673951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.674700 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.674704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.675560 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.675569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.675975 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.675980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.676064 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.676497 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:34.676517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:34.678419 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 16:04:34.678849 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:34.673298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:39441 (local address 127.13.148.62:46081)
0806 16:04:34.673424 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:34.673427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:34.673439 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:04:34.673556 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:34.673558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:34.673595 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:34.673657 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:34.674056 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.674606 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.675670 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.675873 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.675948 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.678607 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:39441: BlockingRecv error: recv got EOF from 127.13.148.1:39441 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 16:04:34.678864 26905 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:04:34.780615 26909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:34.780936 26909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:34.781015 26909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:34.785255 26909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:34.785334 26909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:34.785418 26909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:34.790797 26909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:42873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=44847
--tserver_master_addrs=127.13.148.62:46081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:34.791893 26909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:34.792954 26909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:34.803097 26917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:34.803089 26909 server_base.cc:1034] running on GCE node
W20260806 16:04:34.803110 26915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:34.803066 26914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:34.803750 26909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:34.804404 26909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:34.805612 26909 hybrid_clock.cc:648] HybridClock initialized: now 1786032274805579 us; error 46 us; skew 500 ppm
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032274, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:34.808590 26909 init.cc:377] Logged in from keytab as oryx/127.13.148.2@KRBTEST.COM (short username oryx)
I20260806 16:04:34.809525 26909 webserver.cc:460] Webserver started at http://127.13.148.2:44847/ using document root <none> and password file <none>
I20260806 16:04:34.810086 26909 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:34.810170 26909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:34.814651 26909 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:04:34.817019 26929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.818643 26909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:34.818841 26909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "e6a822e185fc44b2afce1afca9e08605"
format_stamp: "Formatted at 2026-08-06 16:04:33 on dist-test-slave-7kzw"
server_key: "35e147f9a881fa8ff851a19dcb3ea39d"
server_key_iv: "a59b9a99e70badaa3a0b1458c63ea045"
server_key_version: "encryptionkey@0"
I20260806 16:04:34.819434 26909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:34.836412 26909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:34.839993 26909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:34.840217 26909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:34.840858 26909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:34.841722 26909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:34.841780 26909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.841825 26909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:34.841866 26909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:34.851011 26909 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:42873
I20260806 16:04:34.851043 27078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:42873 every 8 connection(s)
I20260806 16:04:34.852059 26909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:34.856357 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 26909
I20260806 16:04:34.856839 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:42035
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:46081
--webserver_port=38515
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.3
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:04:34.864183 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:34.853996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:50873 (local address 127.13.148.62:46081)
0806 16:04:34.854177 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:34.854181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:34.855098 (+ 917us) server_negotiation.cc:407] Connection header received
0806 16:04:34.856477 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:34.856480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:34.856549 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:34.856655 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:34.857946 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.858553 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.859684 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.859911 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.860044 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.864066 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:50873: BlockingRecv error: recv got EOF from 127.13.148.2:50873 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 16:04:34.865131 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:34.854380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:50873)
0806 16:04:34.854926 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:34.854970 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:34.856254 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:34.856845 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:34.856852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:34.857310 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:34.857797 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.857810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.858677 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.858694 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.859549 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.859558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.860147 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.860153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.860324 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.861034 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:34.861051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:34.863936 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":111,"threads_started":1}
W20260806 16:04:34.865670 27079 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:46081 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.62:46081: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:34.871348 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:34.866159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:51553)
0806 16:04:34.866276 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:34.866291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:34.866373 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:34.866661 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:34.866664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:34.866836 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:34.866927 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.866932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.867676 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.867681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.868216 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.868224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.868735 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.868739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.868833 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.869337 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:34.869348 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:34.871213 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 16:04:34.871821 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:34.866281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:51553 (local address 127.13.148.62:46081)
0806 16:04:34.866394 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:34.866400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:34.866417 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:04:34.866509 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:34.866512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:34.866559 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:34.866646 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:34.867042 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.867565 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.868398 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.868606 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.868715 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.871728 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:51553: BlockingRecv error: recv got EOF from 127.13.148.2:51553 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:34.877322 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:34.871927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:39565)
0806 16:04:34.872087 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:34.872099 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:34.872163 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:34.872432 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:34.872435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:34.872600 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:34.872686 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.872692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.873393 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.873397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.874266 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:34.874276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.874772 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.874776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:34.874864 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.875330 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:34.875344 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:34.877181 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260806 16:04:34.877417 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:34.872091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39565 (local address 127.13.148.62:46081)
0806 16:04:34.872214 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:34.872220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:34.872242 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:04:34.872280 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:34.872282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:34.872321 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:34.872407 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:34.872792 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.873269 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.874391 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:34.874662 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:34.874743 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:34.877302 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:39565: BlockingRecv error: recv got EOF from 127.13.148.2:39565 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 16:04:34.877741 27079 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:04:34.974490 27082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:34.974800 27082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:34.974876 27082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:34.978820 27082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:34.978910 27082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:34.978999 27082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:34.984107 27082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:42035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=38515
--tserver_master_addrs=127.13.148.62:46081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:34.985251 27082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:34.986297 27082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:04:34.996382 27082 server_base.cc:1034] running on GCE node
W20260806 16:04:34.996361 27092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:34.996352 27089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:34.996690 27087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:34.997112 27082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:34.997763 27082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:34.998948 27082 hybrid_clock.cc:648] HybridClock initialized: now 1786032274998932 us; error 29 us; skew 500 ppm
Aug 06 16:04:34 dist-test-slave-7kzw krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032274, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:35.002007 27082 init.cc:377] Logged in from keytab as oryx/127.13.148.3@KRBTEST.COM (short username oryx)
I20260806 16:04:35.002969 27082 webserver.cc:460] Webserver started at http://127.13.148.3:38515/ using document root <none> and password file <none>
I20260806 16:04:35.003539 27082 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:35.003620 27082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:35.007575 27082 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:04:35.009709 27102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:35.010962 27082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:04:35.011097 27082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "013a651f991d470db918b1519611615d"
format_stamp: "Formatted at 2026-08-06 16:04:34 on dist-test-slave-7kzw"
server_key: "49015dda87945cba134bc8ecde8f5ade"
server_key_iv: "155575f0fe694610d2a1110b8e0d5078"
server_key_version: "encryptionkey@0"
I20260806 16:04:35.011650 27082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:35.027180 27082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:35.030093 27082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:35.030311 27082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:35.030973 27082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:35.031821 27082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:35.031872 27082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:35.031912 27082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:35.031927 27082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:35.041041 27082 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:42035
I20260806 16:04:35.041074 27265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:42035 every 8 connection(s)
I20260806 16:04:35.042057 27082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:35.045122 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 27082
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 16:04:35.054234 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:35.043941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:55123 (local address 127.13.148.62:46081)
0806 16:04:35.044087 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:35.044091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:35.044964 (+ 873us) server_negotiation.cc:407] Connection header received
0806 16:04:35.046556 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:35.046559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:35.046626 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:35.046699 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:35.048036 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.048633 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.050048 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.050350 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.050459 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.054091 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:55123: BlockingRecv error: recv got EOF from 127.13.148.3:55123 (error 108)
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":12032}
I20260806 16:04:35.054982 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:35.044331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:55123)
0806 16:04:35.044805 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:35.044848 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:35.046276 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:35.046927 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:35.046934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:35.047296 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:35.047884 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.047905 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.048769 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.048773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.049926 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.049936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.050554 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.050558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.050699 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.051275 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:35.051291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:35.053987 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
W20260806 16:04:35.055554 27266 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:46081 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.62:46081: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
I20260806 16:04:35.055665 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:35.046151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.13.148.62:46081)
0806 16:04:35.046482 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:35.046488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:35.046504 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:04:35.046550 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:35.046553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:35.046699 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:35.046806 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:35.047628 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.048324 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.049291 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.049635 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.049752 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.051255 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:35.051261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:35.051274 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:35.051300 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:35.053340 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:35.053814 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:35.053820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:35.053828 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:35.053960 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:35.054263 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:35.054266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:35.054268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:35.054738 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:35.054966 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:35.055264 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 16:04:35.055509 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"spinlock_wait_cycles":4864,"thread_start_us":101,"threads_started":1}
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:35.061086 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:35.056013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:48363)
0806 16:04:35.056137 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:35.056152 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:35.056228 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:35.056437 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:35.056440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:35.056605 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:35.056737 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.056745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.057746 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.057749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.058372 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.058381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.058887 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.058890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.058958 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.059355 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:35.059366 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:35.060968 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 16:04:35.061239 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:35.056120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:48363 (local address 127.13.148.62:46081)
0806 16:04:35.056231 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:35.056235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:35.056247 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:04:35.056301 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:35.056303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:35.056334 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:35.056393 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:35.056858 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.057609 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.058506 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.058785 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.058866 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.061085 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:48363: BlockingRecv error: recv got EOF from 127.13.148.3:48363 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:35.066431 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:35.061806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:43153)
0806 16:04:35.061921 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:35.061934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:35.062011 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:35.062251 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:35.062254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:35.062403 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:35.062511 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.062517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.063208 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.063212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.063729 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.063737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.064168 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.064172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.064249 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.064698 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:35.064712 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:35.066294 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 16:04:35.066519 26907 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:35.061900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:43153 (local address 127.13.148.62:46081)
0806 16:04:35.062062 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:35.062067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:35.062080 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:04:35.062117 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:35.062119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:35.062149 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:35.062234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:35.062628 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.063085 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.063849 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.064059 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.064173 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.066396 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:43153: BlockingRecv error: recv got EOF from 127.13.148.3:43153 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 16:04:35.066713 27266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:35.685338 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:35.679447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:48393)
0806 16:04:35.679824 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:35.679844 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:35.679976 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:35.680266 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:35.680272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:35.680532 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:35.680631 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.680637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.681449 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.681452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.682014 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.682021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.682547 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.682550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.682623 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.683138 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:35.683150 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:35.685213 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":85,"threads_started":1}
W20260806 16:04:35.685532 27274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:35.679563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48393 (local address 127.13.148.62:46081)
0806 16:04:35.679826 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:35.679829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:35.679947 (+ 118us) server_negotiation.cc:407] Connection header received
0806 16:04:35.680079 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:35.680081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:35.680131 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:35.680213 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:35.680790 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.681352 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.682148 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.682376 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.682546 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.685350 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:48393: BlockingRecv error: recv got EOF from 127.13.148.1:48393 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:35 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:35.884509 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:35.878516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:45705)
0806 16:04:35.878789 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:35.878803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:35.878924 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:35.879295 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:35.879298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:35.879512 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:35.879623 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.879632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.880450 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.880453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.881070 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:35.881079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.881569 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.881573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:35.881644 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.882188 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:35.882201 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:35.884338 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
W20260806 16:04:35.884812 27274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:35.878673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:45705 (local address 127.13.148.62:46081)
0806 16:04:35.878811 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:35.878814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:35.878887 (+ 73us) server_negotiation.cc:407] Connection header received
0806 16:04:35.879056 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:35.879060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:35.879120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:35.879198 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:35.879755 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.880297 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.881215 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:35.881432 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:35.881522 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:35.884683 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:45705: BlockingRecv error: recv got EOF from 127.13.148.2:45705 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 16:04:36 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:36 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:36.073634 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:36.067342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:44591)
0806 16:04:36.067619 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:36.067633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:36.067736 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:36.068106 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:36.068109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:36.068306 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:36.068440 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:36.068447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.069291 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.069294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:36.069877 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:36.069886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.070411 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.070418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:36.070575 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:36.071243 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:36.071255 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:36.073499 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
W20260806 16:04:36.073792 27274 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:36.067463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:44591 (local address 127.13.148.62:46081)
0806 16:04:36.067600 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:36.067603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:36.067704 (+ 101us) server_negotiation.cc:407] Connection header received
0806 16:04:36.067858 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:36.067861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:36.067930 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:36.068015 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:36.068574 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.069173 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.070015 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.070269 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.070358 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:36.073631 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:44591: BlockingRecv error: recv got EOF from 127.13.148.3:44591 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 16:04:36 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:36 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:36.691817 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:36.686132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:59721)
0806 16:04:36.686418 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:36.686432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:36.686550 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:36.686819 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:36.686821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:36.687009 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:36.687106 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:36.687111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.687870 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.687873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:36.688430 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:36.688438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.688982 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.688985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:36.689054 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:36.689597 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:36.689610 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:36.691664 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
W20260806 16:04:36.691947 27282 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:36.686177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:59721 (local address 127.13.148.62:46081)
0806 16:04:36.686422 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:36.686433 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:36.686517 (+ 84us) server_negotiation.cc:407] Connection header received
0806 16:04:36.686654 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:36.686656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:36.686701 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:36.686779 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:36.687208 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.687768 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.688579 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.688829 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.688910 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:36.691803 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:59721: BlockingRecv error: recv got EOF from 127.13.148.1:59721 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":63,"threads_started":1}
Aug 06 16:04:36 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:36 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:36.896632 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:36.885453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:34457)
0806 16:04:36.885761 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:36.885800 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:36.885924 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:36.886212 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:36.886215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:36.886449 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:36.886572 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:36.886577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.887357 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.887362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:36.887923 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:36.887931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.888400 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.888404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:36.888492 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:36.894017 (+ 5525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:36.894031 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:36.896471 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
W20260806 16:04:36.896782 27282 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:36.885531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:34457 (local address 127.13.148.62:46081)
0806 16:04:36.885683 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:36.885688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:36.885866 (+ 178us) server_negotiation.cc:407] Connection header received
0806 16:04:36.886039 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:36.886043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:36.886096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:36.886186 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:36.886694 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.887227 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.888063 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:36.888286 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:36.888397 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:36.896629 (+ 8232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:34457: BlockingRecv error: recv got EOF from 127.13.148.2:34457 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 06 16:04:37 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:37 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:37.080447 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:37.074729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:47195)
0806 16:04:37.075037 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:37.075055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:37.075188 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:37.075557 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:37.075560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:37.075769 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:37.075872 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:37.075878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.076634 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.076637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:37.077210 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:37.077220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.077665 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.077667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:37.077762 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:37.078284 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:37.078296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:37.080317 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
W20260806 16:04:37.080736 27282 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:37.074914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:47195 (local address 127.13.148.62:46081)
0806 16:04:37.075061 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:37.075064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:37.075118 (+ 54us) server_negotiation.cc:407] Connection header received
0806 16:04:37.075327 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:37.075330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:37.075390 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:37.075485 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:37.075981 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.076503 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.077337 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.077558 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.077640 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:37.080476 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:47195: BlockingRecv error: recv got EOF from 127.13.148.3:47195 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 16:04:37 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:37 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:37.699451 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:37.692694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:48651)
0806 16:04:37.692978 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:37.692992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:37.693108 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:37.693443 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:37.693446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:37.693668 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:37.693816 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:37.693825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.694868 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.694871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:37.695427 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:37.695436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.696100 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.696104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:37.696176 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:37.696964 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:37.696989 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:37.699268 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
W20260806 16:04:37.699542 27291 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:37.692750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48651 (local address 127.13.148.62:46081)
0806 16:04:37.693091 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:37.693095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:37.693109 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:37.693237 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:37.693241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:37.693295 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:37.693404 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:37.693954 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.694747 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.695566 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.695880 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.695990 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:37.699411 (+ 3421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:48651: BlockingRecv error: recv got EOF from 127.13.148.1:48651 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":119,"threads_started":1}
Aug 06 16:04:37 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:37 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:37.903575 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:37.897562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:56531)
0806 16:04:37.897878 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:37.897893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:37.897994 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:37.898212 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:37.898214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:37.898405 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:37.898535 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:37.898541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.899389 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.899393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:37.900024 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:37.900034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.900528 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.900532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:37.900605 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:37.901177 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:37.901191 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:37.903435 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
W20260806 16:04:37.903909 27291 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:37.897641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:56531 (local address 127.13.148.62:46081)
0806 16:04:37.897819 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:37.897826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:37.897948 (+ 122us) server_negotiation.cc:407] Connection header received
0806 16:04:37.898061 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:37.898063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:37.898105 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:37.898172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:37.898705 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.899244 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.900165 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:37.900405 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:37.900491 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:37.903595 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:56531: BlockingRecv error: recv got EOF from 127.13.148.2:56531 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 16:04:38 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:38 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:38.087659 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:38.081277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:58029)
0806 16:04:38.081698 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:38.081712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:38.081865 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:38.082155 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:38.082158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:38.082384 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:38.082510 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:38.082516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.083327 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.083331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:38.084036 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:38.084046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.084585 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.084590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:38.084717 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:38.085359 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:38.085372 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:38.087456 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":100,"threads_started":1}
W20260806 16:04:38.087693 27291 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:38.081436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58029 (local address 127.13.148.62:46081)
0806 16:04:38.081617 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:38.081623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:38.081797 (+ 174us) server_negotiation.cc:407] Connection header received
0806 16:04:38.081991 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:38.081993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:38.082040 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:38.082128 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:38.082637 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.083200 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.084182 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.084414 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.084558 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:38.087586 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:58029: BlockingRecv error: recv got EOF from 127.13.148.3:58029 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 16:04:38 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:38 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:38.706204 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:38.700455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:60033)
0806 16:04:38.700814 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:38.700832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:38.700937 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:38.701307 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:38.701312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:38.701581 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:38.701725 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:38.701733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.702550 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.702553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:38.703132 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:38.703140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.703570 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.703573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:38.703633 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:38.704172 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:38.704184 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:38.706079 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
W20260806 16:04:38.706398 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:38.700553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:60033 (local address 127.13.148.62:46081)
0806 16:04:38.700874 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:38.700878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:38.700893 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:04:38.701038 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:38.701040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:38.701088 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:38.701243 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:38.701847 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.702391 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.703245 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.703468 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.703551 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:38.706234 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:60033: BlockingRecv error: recv got EOF from 127.13.148.1:60033 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
Aug 06 16:04:38 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:38 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:38.911039 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:38.904871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:50271)
0806 16:04:38.905196 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:38.905219 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:38.905368 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:38.905683 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:38.905686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:38.905996 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:38.906097 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:38.906103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.907085 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.907090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:38.907667 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:38.907682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.908154 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.908158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:38.908238 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:38.908797 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:38.908809 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:38.910902 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":128,"threads_started":1}
W20260806 16:04:38.911167 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:38.905058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:50271 (local address 127.13.148.62:46081)
0806 16:04:38.905202 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:38.905206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:38.905318 (+ 112us) server_negotiation.cc:407] Connection header received
0806 16:04:38.905489 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:38.905492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:38.905550 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:38.905647 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:38.906256 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.906975 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.907802 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:38.908035 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:38.908150 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:38.911044 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:50271: BlockingRecv error: recv got EOF from 127.13.148.2:50271 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 06 16:04:39 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:39 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:39.093966 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:39.088666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:60877)
0806 16:04:39.088945 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:39.088958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:39.089059 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:39.089313 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:39.089316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:39.089507 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:39.089603 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:39.089608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.090353 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.090355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:39.090980 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:39.090989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.091402 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.091405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:39.091464 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:39.092031 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:39.092043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:39.093849 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":104,"threads_started":1}
W20260806 16:04:39.094182 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:39.088781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:60877 (local address 127.13.148.62:46081)
0806 16:04:39.088920 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:39.088923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:39.089029 (+ 106us) server_negotiation.cc:407] Connection header received
0806 16:04:39.089144 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:39.089147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:39.089197 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:39.089269 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:39.089741 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.090261 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.091087 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.091306 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.091385 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:39.094025 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:60877: BlockingRecv error: recv got EOF from 127.13.148.3:60877 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 16:04:39 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:39 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:39.713268 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:39.707217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:59677)
0806 16:04:39.707593 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:39.707612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:39.707737 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:39.707991 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:39.707993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:39.708189 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:39.708299 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:39.708304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.709116 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.709119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:39.709674 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:39.709682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.710109 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.710112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:39.710173 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:39.710863 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:39.710875 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:39.713150 (+ 2275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":295,"threads_started":1}
W20260806 16:04:39.713431 27299 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:39.707385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:59677 (local address 127.13.148.62:46081)
0806 16:04:39.707628 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:39.707631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:39.707679 (+ 48us) server_negotiation.cc:407] Connection header received
0806 16:04:39.707829 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:39.707831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:39.707874 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:39.707956 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:39.708400 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.709017 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.709784 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.710006 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.710088 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:39.713292 (+ 3204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:59677: BlockingRecv error: recv got EOF from 127.13.148.1:59677 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
Aug 06 16:04:39 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:39 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:39.918715 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:39.911992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:39905)
0806 16:04:39.912327 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:39.912342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:39.912438 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:39.912884 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:39.912887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:39.913092 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:39.913251 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:39.913257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.914067 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.914071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:39.914779 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:39.914790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.915455 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.915459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:39.915553 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:39.916284 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:39.916296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:39.918563 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":145,"threads_started":1}
W20260806 16:04:39.918861 27299 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:39.912170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39905 (local address 127.13.148.62:46081)
0806 16:04:39.912341 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:39.912345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:39.912422 (+ 77us) server_negotiation.cc:407] Connection header received
0806 16:04:39.912586 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:39.912589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:39.912707 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:39.912788 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:39.913378 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.913937 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.914925 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:39.915302 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:39.915416 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:39.918729 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:39905: BlockingRecv error: recv got EOF from 127.13.148.2:39905 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 16:04:40 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:40 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:40.100749 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:40.094874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:34333)
0806 16:04:40.095212 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:40.095231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:40.095339 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:40.095633 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:40.095637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:40.095859 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:40.095953 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:40.095959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.096738 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.096742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:40.097324 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:40.097333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.097825 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.097829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:40.097917 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:40.098531 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:40.098545 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:40.100615 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":114,"threads_started":1}
W20260806 16:04:40.101052 27299 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:40.095120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:34333 (local address 127.13.148.62:46081)
0806 16:04:40.095272 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:40.095279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:40.095294 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:04:40.095451 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:40.095454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:40.095510 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:40.095601 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:40.096081 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.096623 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.097452 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.097697 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.097808 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:40.100933 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:34333: BlockingRecv error: recv got EOF from 127.13.148.3:34333 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 16:04:40 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:40 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:40.720171 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:40.714164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:37567)
0806 16:04:40.714492 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:40.714510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:40.714637 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:40.714940 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:40.714943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:40.715234 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:40.715392 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:40.715404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.716259 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.716262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:40.716892 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:40.716900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.717382 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.717385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:40.717451 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:40.717983 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:40.717995 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:40.720035 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
W20260806 16:04:40.720376 27304 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:40.714234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:37567 (local address 127.13.148.62:46081)
0806 16:04:40.714581 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:40.714590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:40.714607 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:04:40.714726 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:40.714729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:40.714788 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:40.714881 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:40.715554 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.716139 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.717024 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.717272 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.717365 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:40.720193 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:37567: BlockingRecv error: recv got EOF from 127.13.148.1:37567 (error 108)
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":68,"threads_started":1}
Aug 06 16:04:40 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:40 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:40.925901 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:40.919614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:54445)
0806 16:04:40.919987 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:40.920003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:40.920168 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:40.920467 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:40.920470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:40.920715 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:40.920812 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:40.920817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.921607 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.921614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:40.922179 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:40.922187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.922767 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.922771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:40.922851 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:40.923512 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:40.923524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:40.925784 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":102,"threads_started":1}
W20260806 16:04:40.926087 27304 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:40.919771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:54445 (local address 127.13.148.62:46081)
0806 16:04:40.919968 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:40.919973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:40.920104 (+ 131us) server_negotiation.cc:407] Connection header received
0806 16:04:40.920300 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:40.920303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:40.920349 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:40.920436 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:40.920935 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.921493 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.922310 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:40.922645 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:40.922777 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:40.925918 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:54445: BlockingRecv error: recv got EOF from 127.13.148.2:54445 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 16:04:41 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:41 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:41.108052 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:41.101557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:41299)
0806 16:04:41.101912 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:41.101936 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:41.102090 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:41.102415 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:41.102419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:41.102703 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:41.102807 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:41.102814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.103709 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.103714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:41.104373 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:41.104382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.104905 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.104909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:41.105005 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:41.105621 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:41.105647 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:41.107872 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":92,"threads_started":1}
W20260806 16:04:41.108121 27304 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:41.101685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:41299 (local address 127.13.148.62:46081)
0806 16:04:41.101873 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:41.101878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:41.102024 (+ 146us) server_negotiation.cc:407] Connection header received
0806 16:04:41.102216 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:41.102220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:41.102282 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:41.102384 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:41.102962 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.103580 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.104513 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.104775 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.104899 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:41.108008 (+ 3109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:41299: BlockingRecv error: recv got EOF from 127.13.148.3:41299 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 16:04:41 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:41 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:41.727339 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:41.721056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:54117)
0806 16:04:41.721598 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:41.721612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:41.721761 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:41.722084 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:41.722087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:41.722329 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:41.722425 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:41.722431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.723360 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.723370 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:41.724058 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:41.724067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.724643 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.724645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:41.724707 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:41.725309 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:41.725320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:41.727209 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":87,"threads_started":1}
W20260806 16:04:41.727546 27310 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:41.721192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:54117 (local address 127.13.148.62:46081)
0806 16:04:41.721492 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:41.721497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:41.721719 (+ 222us) server_negotiation.cc:407] Connection header received
0806 16:04:41.721880 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:41.721884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:41.721941 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:41.722028 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:41.722585 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.723203 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.724197 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.724529 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.724628 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:41.727383 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:54117: BlockingRecv error: recv got EOF from 127.13.148.1:54117 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":67,"threads_started":1}
Aug 06 16:04:41 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:41 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:41.937526 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:41.926797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:54167)
0806 16:04:41.927163 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:41.927179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:41.927303 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:41.927764 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:41.927767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:41.927985 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:41.928088 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:41.928094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.928882 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.928885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:41.929510 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:41.929520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.930023 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.930026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:41.930097 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:41.934681 (+ 4584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:41.934696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:41.937350 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":140,"threads_started":1}
W20260806 16:04:41.937608 27310 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:41.926872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:54167 (local address 127.13.148.62:46081)
0806 16:04:41.927065 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:41.927070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:41.927260 (+ 190us) server_negotiation.cc:407] Connection header received
0806 16:04:41.927467 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:41.927470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:41.927520 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:41.927657 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:41.928214 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.928768 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.929666 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:41.929897 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:41.929980 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:41.937493 (+ 7513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:54167: BlockingRecv error: recv got EOF from 127.13.148.2:54167 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 16:04:42 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:42 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:42.117153 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:42.109308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:45361)
0806 16:04:42.109869 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:42.109888 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:42.110016 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:42.110351 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:42.110357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:42.110648 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:42.110773 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:42.110800 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.112057 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.112060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:42.112840 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:42.112851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.113466 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.113473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:42.113591 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:42.114422 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:42.114443 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:42.116950 (+ 2507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":96,"threads_started":1}
W20260806 16:04:42.117249 27310 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:42.109390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:45361 (local address 127.13.148.62:46081)
0806 16:04:42.109547 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:42.109551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:42.110047 (+ 496us) server_negotiation.cc:407] Connection header received
0806 16:04:42.110121 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:42.110127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:42.110195 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:42.110298 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:42.110913 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.111932 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.112962 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.113330 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.113447 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:42.117114 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:45361: BlockingRecv error: recv got EOF from 127.13.148.3:45361 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 16:04:42 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:42 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:42.733945 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:42.728170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:60171)
0806 16:04:42.728450 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:42.728463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:42.728551 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:42.728825 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:42.728827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:42.729019 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:42.729117 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:42.729123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.729908 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.729911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:42.730515 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:42.730529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.731103 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.731107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:42.731175 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:42.731757 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:42.731770 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:42.733759 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
W20260806 16:04:42.734018 27314 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:42.728366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:60171 (local address 127.13.148.62:46081)
0806 16:04:42.728609 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:42.728613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:42.728625 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:04:42.728660 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:42.728662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:42.728703 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:42.728785 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:42.729239 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.729811 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.730674 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.730913 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.731014 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:42.733893 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:60171: BlockingRecv error: recv got EOF from 127.13.148.1:60171 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
Aug 06 16:04:42 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:42 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:42.944761 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:42.938507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:47303)
0806 16:04:42.938835 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:42.938849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:42.938954 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:42.939305 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:42.939308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:42.939502 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:42.939597 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:42.939602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.940674 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.940681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:42.941389 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:42.941400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.942005 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.942009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:42.942092 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:42.942623 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:42.942635 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:42.944637 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
W20260806 16:04:42.945057 27314 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:42.938704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:47303 (local address 127.13.148.62:46081)
0806 16:04:42.938852 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:42.938856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:42.938921 (+ 65us) server_negotiation.cc:407] Connection header received
0806 16:04:42.939087 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:42.939090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:42.939168 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:42.939258 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:42.939745 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.940558 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.941550 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:42.941882 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:42.942000 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:42.944912 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:47303: BlockingRecv error: recv got EOF from 127.13.148.2:47303 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 16:04:43 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:43 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:43.124076 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:43.118027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:45677)
0806 16:04:43.118329 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:43.118347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:43.118514 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:43.118988 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:43.118991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:43.119201 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:43.119293 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:43.119299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.120161 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.120166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:43.120721 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:43.120728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.121191 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.121194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:43.121279 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:43.121846 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:43.121859 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:43.123942 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":91,"threads_started":1}
W20260806 16:04:43.124423 27314 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:43.118132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:45677 (local address 127.13.148.62:46081)
0806 16:04:43.118321 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:43.118327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:43.118434 (+ 107us) server_negotiation.cc:407] Connection header received
0806 16:04:43.118629 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:43.118632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:43.118734 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:43.118839 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:43.119419 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.120047 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.120844 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.121068 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.121181 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:43.124099 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:45677: BlockingRecv error: recv got EOF from 127.13.148.3:45677 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 06 16:04:43 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:43 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:43.741667 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:43.734930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:49835)
0806 16:04:43.735226 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:43.735244 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:43.735358 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:43.735612 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:43.735615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:43.735851 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:43.735954 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:43.735960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.736817 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.736821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:43.737709 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:43.737723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.738298 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.738302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:43.738419 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:43.738993 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:43.739006 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:43.741453 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
W20260806 16:04:43.741807 27319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:43.735057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:49835 (local address 127.13.148.62:46081)
0806 16:04:43.735348 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:43.735351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:43.735366 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:04:43.735438 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:43.735441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:43.735481 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:43.735564 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:43.736074 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.736667 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.737891 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.738165 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.738294 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:43.741651 (+ 3357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:49835: BlockingRecv error: recv got EOF from 127.13.148.1:49835 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":130,"threads_started":1}
Aug 06 16:04:43 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:43 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:43.951623 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:43.945591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:54875)
0806 16:04:43.945872 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:43.945886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:43.945984 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:43.946233 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:43.946235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:43.946426 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:43.946597 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:43.946603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.947407 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.947409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:43.947962 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:43.947971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.948551 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.948554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:43.948621 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:43.949246 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:43.949258 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:43.951429 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
W20260806 16:04:43.951727 27319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:43.945756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:54875 (local address 127.13.148.62:46081)
0806 16:04:43.945932 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:43.945936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:43.945950 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:43.946074 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:43.946076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:43.946120 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:43.946191 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:43.946738 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.947299 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.948114 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:43.948423 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:43.948510 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:43.951587 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:54875: BlockingRecv error: recv got EOF from 127.13.148.2:54875 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 16:04:44 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:44 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:44.131227 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:44.124877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:47725)
0806 16:04:44.125196 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:44.125211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:44.125319 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:44.125655 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:44.125658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:44.125883 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:44.125982 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:44.125988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.127066 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.127069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:44.127673 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:44.127715 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.128278 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.128284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:44.128381 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:44.129019 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:44.129031 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:44.131105 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":120,"threads_started":1}
W20260806 16:04:44.131434 27319 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:44.124943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:47725 (local address 127.13.148.62:46081)
0806 16:04:44.125094 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:44.125099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:44.125311 (+ 212us) server_negotiation.cc:407] Connection header received
0806 16:04:44.125394 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:44.125397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:44.125449 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:44.125533 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:44.126111 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.126936 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.127840 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.128124 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.128254 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:44.131260 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:47725: BlockingRecv error: recv got EOF from 127.13.148.3:47725 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 16:04:44 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:44 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:44.748989 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:44.742597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:38729)
0806 16:04:44.742894 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:44.742909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:44.742997 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:44.743320 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:44.743322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:44.743522 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:44.743620 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:44.743626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.744521 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.744524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:44.745103 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:44.745112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.745630 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.745633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:44.745720 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:44.746282 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:44.746295 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:44.748817 (+ 2522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
W20260806 16:04:44.749127 27329 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:44.742762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38729 (local address 127.13.148.62:46081)
0806 16:04:44.743061 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:44.743064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:44.743078 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:44.743127 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:44.743129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:44.743174 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:44.743269 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:44.743802 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.744399 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.745256 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.745514 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.745652 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:44.748987 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:38729: BlockingRecv error: recv got EOF from 127.13.148.1:38729 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":123,"threads_started":1}
Aug 06 16:04:44 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:44 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:44.959318 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:44.952561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:45423)
0806 16:04:44.952872 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:44.952895 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:44.952998 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:44.953275 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:44.953277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:44.953475 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:44.953580 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:44.953586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.954657 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.954660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:44.955588 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:44.955610 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.956294 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.956301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:44.956381 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:44.957002 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:44.957016 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:44.959195 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":139,"threads_started":1}
W20260806 16:04:44.959471 27329 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:44.952735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:45423 (local address 127.13.148.62:46081)
0806 16:04:44.952892 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:44.952895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:44.953003 (+ 108us) server_negotiation.cc:407] Connection header received
0806 16:04:44.953065 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:44.953067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:44.953120 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:44.953210 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:44.953873 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.954453 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.955768 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:44.956140 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:44.956261 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:44.959337 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:45423: BlockingRecv error: recv got EOF from 127.13.148.2:45423 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 16:04:45 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:45 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:45.138559 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:45.132203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:32873)
0806 16:04:45.132532 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:45.132553 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:45.132664 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:45.133107 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:45.133110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:45.133362 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:45.133484 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:45.133493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.134505 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.134508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:45.135273 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:45.135285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.135772 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.135775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:45.135841 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:45.136403 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:45.136416 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:45.138361 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":75,"threads_started":1}
W20260806 16:04:45.138787 27329 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:45.132495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:32873 (local address 127.13.148.62:46081)
0806 16:04:45.132694 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:45.132698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:45.132732 (+ 34us) server_negotiation.cc:407] Connection header received
0806 16:04:45.132774 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:45.132777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:45.132821 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:45.132884 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:45.133692 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.134348 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.135386 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.135638 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.135723 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:45.138545 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:32873: BlockingRecv error: recv got EOF from 127.13.148.3:32873 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Aug 06 16:04:45 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:45 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:45.755790 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:45.749947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:48339)
0806 16:04:45.750268 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:45.750282 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:45.750384 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:45.750705 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:45.750707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:45.750910 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:45.751016 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:45.751022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.751877 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.751881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:45.752439 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:45.752448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.753020 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.753023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:45.753090 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:45.753709 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:45.753721 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:45.755672 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
W20260806 16:04:45.756017 27336 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:45.750094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48339 (local address 127.13.148.62:46081)
0806 16:04:45.750348 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:45.750351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:45.750365 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:45.750499 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:45.750502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:45.750545 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:45.750666 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:45.751167 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.751737 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.752615 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.752878 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.752986 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:45.755866 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:48339: BlockingRecv error: recv got EOF from 127.13.148.1:48339 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":65,"threads_started":1}
Aug 06 16:04:45 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:45 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:45.966225 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:45.960122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:57839)
0806 16:04:45.960500 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:45.960517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:45.960704 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:45.960988 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:45.960991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:45.961235 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:45.961336 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:45.961342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.962131 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.962136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:45.962756 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:45.962765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.963250 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.963254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:45.963340 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:45.963900 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:45.963911 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:45.966054 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":161,"threads_started":1}
W20260806 16:04:45.966444 27336 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:45.960302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:57839 (local address 127.13.148.62:46081)
0806 16:04:45.960492 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:45.960497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:45.960637 (+ 140us) server_negotiation.cc:407] Connection header received
0806 16:04:45.960810 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:45.960813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:45.960868 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:45.960961 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:45.961466 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.962005 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.962884 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:45.963119 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:45.963233 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:45.966248 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:57839: BlockingRecv error: recv got EOF from 127.13.148.2:57839 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 16:04:46 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:46 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:46.145773 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:46.139420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:37459)
0806 16:04:46.139729 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:46.139745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:46.139863 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:46.140216 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:46.140219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:46.140420 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:46.140519 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:46.140524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.141431 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.141434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:46.142026 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:46.142035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.142623 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.142627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:46.142705 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:46.143241 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:46.143253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:46.145537 (+ 2284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
W20260806 16:04:46.145938 27336 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:46.139572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:37459 (local address 127.13.148.62:46081)
0806 16:04:46.139764 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:46.139768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:46.139830 (+ 62us) server_negotiation.cc:407] Connection header received
0806 16:04:46.140005 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:46.140008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:46.140061 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:46.140149 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:46.140712 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.141281 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.142157 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.142396 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.142521 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:46.145769 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:37459: BlockingRecv error: recv got EOF from 127.13.148.3:37459 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 16:04:46 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:46 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:46.763092 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:46.756662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:54807)
0806 16:04:46.757064 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:46.757086 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:46.757260 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:46.757587 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:46.757591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:46.757799 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:46.757909 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:46.757916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.758781 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.758784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:46.759420 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:46.759429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.759929 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.759932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:46.760018 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:46.760539 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:46.760552 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:46.762912 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":120,"threads_started":1}
W20260806 16:04:46.763237 27342 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:46.756820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:54807 (local address 127.13.148.62:46081)
0806 16:04:46.757092 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:46.757096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:46.757185 (+ 89us) server_negotiation.cc:407] Connection header received
0806 16:04:46.757360 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:46.757363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:46.757409 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:46.757487 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:46.758044 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.758619 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.759545 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.759781 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.759867 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:46.763076 (+ 3209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:54807: BlockingRecv error: recv got EOF from 127.13.148.1:54807 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
Aug 06 16:04:46 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:46 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:46.978147 27343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:46.967215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:42931)
0806 16:04:46.967528 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:46.967541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:46.967641 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:46.968006 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:46.968010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:46.968236 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:46.968366 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:46.968373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.969661 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.969667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:46.970302 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:46.970314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.971053 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.971057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:46.971144 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:46.975693 (+ 4549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:46.975707 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:46.978012 (+ 2305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":116,"threads_started":1}
W20260806 16:04:46.978448 27342 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:46.967344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:42931 (local address 127.13.148.62:46081)
0806 16:04:46.967525 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:46.967530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:46.967600 (+ 70us) server_negotiation.cc:407] Connection header received
0806 16:04:46.967751 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:46.967756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:46.967823 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:46.967932 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:46.968502 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.969509 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.970449 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:46.970925 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:46.971067 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:46.978275 (+ 7208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:42931: BlockingRecv error: recv got EOF from 127.13.148.2:42931 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 16:04:47 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:47 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:47.152544 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:47.146781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:50703)
0806 16:04:47.147057 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:47.147071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:47.147177 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:47.147474 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:47.147477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:47.147740 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:47.147872 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:47.147879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.148696 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.148699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:47.149305 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:47.149313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.149750 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.149752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:47.149814 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:47.150347 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:47.150359 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:47.152415 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
W20260806 16:04:47.152812 27342 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:47.146843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50703 (local address 127.13.148.62:46081)
0806 16:04:47.146985 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:47.146989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:47.147148 (+ 159us) server_negotiation.cc:407] Connection header received
0806 16:04:47.147259 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:47.147261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:47.147317 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:47.147389 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:47.148023 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.148577 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.149422 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.149646 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.149728 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:47.152682 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:50703: BlockingRecv error: recv got EOF from 127.13.148.3:50703 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 16:04:47 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:47 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:47.770160 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:47.764066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:45829)
0806 16:04:47.764369 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:47.764383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:47.764476 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:47.764817 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:47.764819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:47.765012 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:47.765173 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:47.765182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.766196 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.766200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:47.766934 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:47.766946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.767442 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.767445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:47.767520 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:47.768061 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:47.768072 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:47.770015 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
W20260806 16:04:47.770377 27348 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:47.764242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:45829 (local address 127.13.148.62:46081)
0806 16:04:47.764508 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:47.764512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:47.764526 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:47.764568 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:47.764570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:47.764647 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:47.764779 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:47.765310 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.766070 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.767077 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.767324 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.767416 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:47.770190 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:45829: BlockingRecv error: recv got EOF from 127.13.148.1:45829 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
Aug 06 16:04:47 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:47 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:47.985402 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:47.979156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:49561)
0806 16:04:47.979465 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:47.979480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:47.979618 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:47.979916 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:47.979920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:47.980167 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:47.980269 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:47.980275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.981123 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.981133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:47.981727 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:47.981736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.982239 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.982243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:47.982336 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:47.982920 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:47.982933 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:47.985214 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":125,"threads_started":1}
W20260806 16:04:47.985476 27348 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:47.979465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:49561 (local address 127.13.148.62:46081)
0806 16:04:47.979631 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:47.979636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:47.979653 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:04:47.979706 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:47.979713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:47.979768 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:47.979864 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:47.980402 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.980988 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.981858 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:47.982111 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:47.982233 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:47.985373 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:49561: BlockingRecv error: recv got EOF from 127.13.148.2:49561 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 16:04:48 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:48 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:48.160142 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:48.153380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:42535)
0806 16:04:48.153967 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:48.153981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:48.154126 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:48.154415 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:48.154418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:48.154675 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:48.154777 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:48.154783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.155565 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.155570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:48.156154 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:48.156163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.156633 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.156637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:48.156738 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:48.157325 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:48.157336 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:48.159925 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":109,"threads_started":1}
W20260806 16:04:48.160211 27348 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:48.153617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:42535 (local address 127.13.148.62:46081)
0806 16:04:48.153910 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:48.153915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:48.154064 (+ 149us) server_negotiation.cc:407] Connection header received
0806 16:04:48.154239 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:48.154243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:48.154298 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:48.154389 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:48.154909 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.155447 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.156281 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.156508 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.156618 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:48.160078 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:42535: BlockingRecv error: recv got EOF from 127.13.148.3:42535 (error 108)
Metrics: {"server-negotiator.queue_time_us":190}
Aug 06 16:04:48 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:48 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:48.777086 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:48.771001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:58285)
0806 16:04:48.771279 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:48.771294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:48.771384 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:48.771634 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:48.771636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:48.771881 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:48.771989 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:48.771994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.772882 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.772886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:48.773585 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:48.773596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.774131 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.774134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:48.774210 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:48.774793 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:48.774805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:48.776962 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
W20260806 16:04:48.777292 27353 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:48.771041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:58285 (local address 127.13.148.62:46081)
0806 16:04:48.771348 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:48.771352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:48.771366 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:04:48.771470 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:48.771472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:48.771517 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:48.771596 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:48.772143 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.772742 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.773729 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.773961 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.774045 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:48.777117 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:58285: BlockingRecv error: recv got EOF from 127.13.148.1:58285 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
Aug 06 16:04:48 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:48 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:48.992552 27354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:48.986316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:34529)
0806 16:04:48.986633 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:48.986653 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:48.986817 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:48.987101 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:48.987104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:48.987392 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:48.987557 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:48.987566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.988437 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.988441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:48.989008 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:48.989016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.989482 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.989485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:48.989571 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:48.990152 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:48.990164 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:48.992377 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
W20260806 16:04:48.992666 27353 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:48.986429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:34529 (local address 127.13.148.62:46081)
0806 16:04:48.986628 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:48.986633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:48.986774 (+ 141us) server_negotiation.cc:407] Connection header received
0806 16:04:48.986927 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:48.986930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:48.986978 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:48.987061 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:48.987711 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.988311 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.989138 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:48.989362 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:48.989474 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:48.992511 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:34529: BlockingRecv error: recv got EOF from 127.13.148.2:34529 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 06 16:04:49 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:49 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:49.167443 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:49.161095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.3:59745)
0806 16:04:49.161488 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:49.161512 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:49.161676 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:49.161989 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:49.161992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:49.162238 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:49.162342 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:49.162349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.163226 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.163233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:49.163836 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:49.163846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.164348 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.164352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:49.164449 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:49.165100 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:49.165114 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:49.167235 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":93,"threads_started":1}
W20260806 16:04:49.167510 27353 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:49.161256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:59745 (local address 127.13.148.62:46081)
0806 16:04:49.161447 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:49.161454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:49.161608 (+ 154us) server_negotiation.cc:407] Connection header received
0806 16:04:49.161791 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:49.161796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:49.161861 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:49.161961 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:49.162507 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.163093 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.163973 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.164219 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.164344 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:49.167389 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:59745: BlockingRecv error: recv got EOF from 127.13.148.3:59745 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 06 16:04:49 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:49 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:49.784839 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:49.778269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.1:58461)
0806 16:04:49.778604 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:49.778654 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:49.778756 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:49.779033 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:49.779035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:49.779246 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:49.779347 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:49.779354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.780322 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.780326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:49.780973 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:49.780983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.781505 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.781509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:49.781619 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:49.782312 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:49.782325 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:49.784642 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":77,"threads_started":1}
W20260806 16:04:49.784950 27358 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:49.778321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:58461 (local address 127.13.148.62:46081)
0806 16:04:49.778638 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:49.778642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:49.778709 (+ 67us) server_negotiation.cc:407] Connection header received
0806 16:04:49.778855 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:49.778858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:49.778906 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:49.778995 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:49.779475 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.780169 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.781130 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.781375 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.781466 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:49.784814 (+ 3348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:58461: BlockingRecv error: recv got EOF from 127.13.148.1:58461 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":138,"threads_started":1}
Aug 06 16:04:49 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 16:04:49 dist-test-slave-7kzw krb5kdc[25880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 16:04:50.000324 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:49.993667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46081 (local address 127.13.148.2:34341)
0806 16:04:49.993944 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:49.993964 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:49.994071 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:49.994446 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:49.994450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:49.994681 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:49.994849 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:49.994858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.995829 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.995852 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:49.996782 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:49.996795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.997280 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.997283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:49.997353 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:49.998018 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:49.998046 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:50.000191 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:46081: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
W20260806 16:04:50.000547 27358 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:49.993725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:34341 (local address 127.13.148.62:46081)
0806 16:04:49.993984 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:49.993988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:49.994021 (+ 33us) server_negotiation.cc:407] Connection header received
0806 16:04:49.994209 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:49.994212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:49.994276 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:49.994374 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:49.995036 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.995672 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.996918 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:49.997154 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:49.997238 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:50.000397 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:34341: BlockingRecv error: recv got EOF from 127.13.148.2:34341 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260806 16:04:50.056394 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 26697
I20260806 16:04:50.070132 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 26909
I20260806 16:04:50.082540 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 27082
I20260806 16:04:50.097177 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 26603
2026-08-06T16:04:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18347 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T16:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:50Z Disabled control of system clock
I20260806 16:04:50.139750 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45653
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:45653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 16:04:50.265522 27369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:50.265827 27369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:50.265865 27369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:50.270116 27369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:50.270208 27369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:50.270224 27369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:50.270241 27369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:50.270256 27369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:50.275883 27369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:45653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:50.277560 27369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:50.279147 27369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:04:50.288954 27369 server_base.cc:1034] running on GCE node
W20260806 16:04:50.288864 27375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:50.288861 27376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:50.288861 27378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:50.289906 27369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:50.291311 27369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:50.292537 27369 hybrid_clock.cc:648] HybridClock initialized: now 1786032290292512 us; error 48 us; skew 500 ppm
I20260806 16:04:50.294854 27369 webserver.cc:460] Webserver started at http://127.13.148.62:41013/ using document root <none> and password file <none>
I20260806 16:04:50.295512 27369 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:50.295661 27369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:50.295953 27369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:50.298172 27369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "91903c78a1414a30afb43b4b985ce132"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "ab78595e3afaec8f3b35f17e01e03b87"
server_key_iv: "a59e0b89b75f3b84eef38abdb31131ca"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.298837 27369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "91903c78a1414a30afb43b4b985ce132"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "ab78595e3afaec8f3b35f17e01e03b87"
server_key_iv: "a59e0b89b75f3b84eef38abdb31131ca"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.302796 27369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:50.305742 27390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.307542 27369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:50.307734 27369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "91903c78a1414a30afb43b4b985ce132"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "ab78595e3afaec8f3b35f17e01e03b87"
server_key_iv: "a59e0b89b75f3b84eef38abdb31131ca"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.307880 27369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:50.321719 27369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:50.322608 27369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:50.322868 27369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:50.330557 27369 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:45653
I20260806 16:04:50.330576 27464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:45653 every 8 connection(s)
I20260806 16:04:50.331561 27369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:50.335018 27465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:50.338037 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 27369
I20260806 16:04:50.338178 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:50.338539 13904 external_mini_cluster.cc:1428] Setting key 8152737410d0c6a5111fdb542bca11ad
I20260806 16:04:50.341719 27465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132: Bootstrap starting.
I20260806 16:04:50.344113 27465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:50.344884 27465 log.cc:824] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:50.346979 27465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132: No bootstrap required, opened a new log
I20260806 16:04:50.348264 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:50.340089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.13.148.62:45653)
0806 16:04:50.340881 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:50.340898 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:50.340938 (+ 40us) server_negotiation.cc:407] Connection header received
0806 16:04:50.341569 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:50.341587 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:50.342062 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:50.342411 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:50.342887 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.343866 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.344834 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.345354 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.345528 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:50.346451 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:50.346456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:50.346520 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:50.346925 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:50.346983 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:50.347004 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:50.347118 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:50.347302 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 16:04:50.347569 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":149,"threads_started":1}
I20260806 16:04:50.350726 27465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91903c78a1414a30afb43b4b985ce132" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45653 } }
I20260806 16:04:50.350970 27465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:50.351049 27465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91903c78a1414a30afb43b4b985ce132, State: Initialized, Role: FOLLOWER
I20260806 16:04:50.351459 27465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91903c78a1414a30afb43b4b985ce132" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45653 } }
I20260806 16:04:50.351610 27465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:50.351706 27465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:50.351830 27465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:50.352762 27465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91903c78a1414a30afb43b4b985ce132" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45653 } }
I20260806 16:04:50.353102 27465 leader_election.cc:302] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91903c78a1414a30afb43b4b985ce132; no voters:
I20260806 16:04:50.353412 27465 leader_election.cc:288] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:50.353562 27473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:50.353826 27473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [term 1 LEADER]: Becoming Leader. State: Replica: 91903c78a1414a30afb43b4b985ce132, State: Running, Role: LEADER
I20260806 16:04:50.354215 27473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91903c78a1414a30afb43b4b985ce132" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45653 } }
I20260806 16:04:50.354703 27465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:50.355890 27476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91903c78a1414a30afb43b4b985ce132. Latest consensus state: current_term: 1 leader_uuid: "91903c78a1414a30afb43b4b985ce132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91903c78a1414a30afb43b4b985ce132" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45653 } } }
I20260806 16:04:50.356020 27476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:50.356438 27474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91903c78a1414a30afb43b4b985ce132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91903c78a1414a30afb43b4b985ce132" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45653 } } }
I20260806 16:04:50.356604 27474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:50.356810 27486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:50.359645 27486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:50.365183 27486 catalog_manager.cc:1347] Generated new cluster ID: 1ad2a96dfc78463482e57533bbca0d42
I20260806 16:04:50.365267 27486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:50.383363 27486 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:50.384358 27486 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:50.392516 27486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91903c78a1414a30afb43b4b985ce132: Generated new TSK 0
I20260806 16:04:50.393298 27486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:04:50.401901 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45653
--builtin_ntp_servers=127.13.148.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 16:04:50.521863 27500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:50.522135 27500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:50.522172 27500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:50.526206 27500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:50.526275 27500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:50.526329 27500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:50.531350 27500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:50.532500 27500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:50.533921 27500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:50.543788 27512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:50.543792 27510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:50.543958 27509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:50.543865 27500 server_base.cc:1034] running on GCE node
I20260806 16:04:50.544461 27500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:50.545104 27500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:50.546300 27500 hybrid_clock.cc:648] HybridClock initialized: now 1786032290546264 us; error 47 us; skew 500 ppm
I20260806 16:04:50.548386 27500 webserver.cc:460] Webserver started at http://127.13.148.1:32851/ using document root <none> and password file <none>
I20260806 16:04:50.549008 27500 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:50.549081 27500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:50.549309 27500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:50.551501 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "d41a41c3fa984305b8e19bd87dd75174"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "5bf6513ba968ca7abcc8f5b45f37cd27"
server_key_iv: "a1cf8cb77b5ee4e94444957a440a70f8"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.552075 27500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "d41a41c3fa984305b8e19bd87dd75174"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "5bf6513ba968ca7abcc8f5b45f37cd27"
server_key_iv: "a1cf8cb77b5ee4e94444957a440a70f8"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.555954 27500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:50.558626 27522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.559995 27500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:04:50.560125 27500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "d41a41c3fa984305b8e19bd87dd75174"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "5bf6513ba968ca7abcc8f5b45f37cd27"
server_key_iv: "a1cf8cb77b5ee4e94444957a440a70f8"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.560252 27500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:50.574038 27500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:50.574915 27500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:50.575131 27500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:50.575766 27500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:50.576735 27500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:50.576784 27500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.576853 27500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:50.576886 27500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.586222 27500 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:39793
I20260806 16:04:50.586252 27671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:39793 every 8 connection(s)
I20260806 16:04:50.587143 27500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:50.590404 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 27500
I20260806 16:04:50.590559 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:50.590871 13904 external_mini_cluster.cc:1428] Setting key 71dc7b118342e05096e2df9e751de70d
I20260806 16:04:50.592871 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45653
--builtin_ntp_servers=127.13.148.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 16:04:50.597622 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:50.589058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:36911 (local address 127.13.148.62:45653)
0806 16:04:50.589221 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:50.589225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:50.590293 (+ 1068us) server_negotiation.cc:407] Connection header received
0806 16:04:50.592097 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:50.592100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:50.592159 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:50.592270 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:50.593226 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.593797 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.595384 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.595660 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.595747 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:50.597081 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:50.597086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:50.597094 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:50.597106 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:50.597145 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:50.597153 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:50.597240 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:50.597412 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 16:04:50.597477 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:04:50.598604 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:50.589447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45653 (local address 127.13.148.1:36911)
0806 16:04:50.590125 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:50.590169 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:50.591845 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:50.592455 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:50.592462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:50.592552 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:50.593091 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:50.593109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.593936 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.593940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:50.595228 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:50.595244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.595837 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.595842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:50.596063 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:50.596792 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:50.596794 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:50.596836 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:50.596903 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:50.597243 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:50.597255 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:50.597263 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:04:50.597412 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 16:04:50.597756 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260806 16:04:50.599982 27672 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45653
I20260806 16:04:50.600286 27672 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:50.600822 27672 heartbeater.cc:511] Master 127.13.148.62:45653 requested a full tablet report, sending...
I20260806 16:04:50.602630 27409 ts_manager.cc:194] Registered new tserver with Master: d41a41c3fa984305b8e19bd87dd75174 (127.13.148.1:39793)
I20260806 16:04:50.604439 27409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.1:36911
W20260806 16:04:50.715726 27676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:50.716090 27676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:50.716135 27676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:50.720798 27676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:50.720872 27676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:50.720934 27676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:50.726167 27676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:50.727396 27676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:50.728960 27676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:50.739078 27681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:50.739018 27682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:50.739110 27676 server_base.cc:1034] running on GCE node
W20260806 16:04:50.739037 27684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:50.739709 27676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:50.740511 27676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:50.741710 27676 hybrid_clock.cc:648] HybridClock initialized: now 1786032290741682 us; error 41 us; skew 500 ppm
I20260806 16:04:50.743809 27676 webserver.cc:460] Webserver started at http://127.13.148.2:33515/ using document root <none> and password file <none>
I20260806 16:04:50.744427 27676 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:50.744529 27676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:50.744779 27676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:50.746977 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "a1a7b730d9ef4c058c1af61e95da76af"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "7b8da6c15fdc010f63ca5da3807a97ec"
server_key_iv: "e184e60b714ff71ba9c9654769a4d2a5"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.747622 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "a1a7b730d9ef4c058c1af61e95da76af"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "7b8da6c15fdc010f63ca5da3807a97ec"
server_key_iv: "e184e60b714ff71ba9c9654769a4d2a5"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.751530 27676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:50.754194 27692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.755532 27676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 16:04:50.755692 27676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "a1a7b730d9ef4c058c1af61e95da76af"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "7b8da6c15fdc010f63ca5da3807a97ec"
server_key_iv: "e184e60b714ff71ba9c9654769a4d2a5"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.755836 27676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:50.769217 27676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:50.770033 27676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:50.770275 27676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:50.771056 27676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:50.772137 27676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:50.772219 27676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.772300 27676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:50.772359 27676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.782698 27676 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:38893
I20260806 16:04:50.782719 27849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:38893 every 8 connection(s)
I20260806 16:04:50.783756 27676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:50.791301 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 27676
I20260806 16:04:50.791433 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:50.791733 13904 external_mini_cluster.cc:1428] Setting key 51a78ceb75f62b2549e07789aa50bdc6
I20260806 16:04:50.794251 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45653
--builtin_ntp_servers=127.13.148.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 16:04:50.794816 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:50.785919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:35501 (local address 127.13.148.62:45653)
0806 16:04:50.786107 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:50.786113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:50.787383 (+ 1270us) server_negotiation.cc:407] Connection header received
0806 16:04:50.788816 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:50.788819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:50.788873 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:50.788960 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:50.789904 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.790593 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.792186 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.792507 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.792599 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:50.794062 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:50.794065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:50.794067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:50.794079 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:50.794100 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:50.794108 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:50.794185 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:50.794574 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 16:04:50.794676 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 16:04:50.795879 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:50.786392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45653 (local address 127.13.148.2:35501)
0806 16:04:50.787195 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:50.787237 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:50.788582 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:50.789128 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:50.789135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:50.789225 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:50.789764 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:50.789778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.790719 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.790722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:50.792022 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:50.792034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.792683 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.792687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:50.792840 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:50.793641 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:50.793644 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:50.793700 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:50.793787 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:50.794222 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:50.794233 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:50.794240 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:04:50.794398 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 16:04:50.794790 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":153,"threads_started":1}
I20260806 16:04:50.797191 27851 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45653
I20260806 16:04:50.797472 27851 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:50.798110 27851 heartbeater.cc:511] Master 127.13.148.62:45653 requested a full tablet report, sending...
I20260806 16:04:50.799384 27409 ts_manager.cc:194] Registered new tserver with Master: a1a7b730d9ef4c058c1af61e95da76af (127.13.148.2:38893)
I20260806 16:04:50.800210 27409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.2:35501
W20260806 16:04:50.916332 27854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:50.916635 27854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:50.916676 27854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:50.920867 27854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:50.920943 27854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:50.920998 27854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:50.926328 27854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:50.927634 27854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:50.929203 27854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:50.940145 27862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:50.940227 27854 server_base.cc:1034] running on GCE node
W20260806 16:04:50.940155 27860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:50.940167 27864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:50.941129 27854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:50.941898 27854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:50.943141 27854 hybrid_clock.cc:648] HybridClock initialized: now 1786032290943103 us; error 49 us; skew 500 ppm
I20260806 16:04:50.945464 27854 webserver.cc:460] Webserver started at http://127.13.148.3:41991/ using document root <none> and password file <none>
I20260806 16:04:50.946152 27854 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:50.946264 27854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:50.946606 27854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:50.949041 27854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "8f1f75a3b1d94b3596479701b8c0dca8"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "6106fa85a76d9ff62c84d9d1e0fe05ba"
server_key_iv: "14f280cca8006eb30779860f3e7cb0d6"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.949760 27854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "8f1f75a3b1d94b3596479701b8c0dca8"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "6106fa85a76d9ff62c84d9d1e0fe05ba"
server_key_iv: "14f280cca8006eb30779860f3e7cb0d6"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.954386 27854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 16:04:50.957633 27873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.959288 27854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 16:04:50.959461 27854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "8f1f75a3b1d94b3596479701b8c0dca8"
format_stamp: "Formatted at 2026-08-06 16:04:50 on dist-test-slave-7kzw"
server_key: "6106fa85a76d9ff62c84d9d1e0fe05ba"
server_key_iv: "14f280cca8006eb30779860f3e7cb0d6"
server_key_version: "encryptionkey@0"
I20260806 16:04:50.959607 27854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:50.977880 27854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:50.978740 27854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:50.978951 27854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:50.979590 27854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:50.980633 27854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:50.980719 27854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.980813 27854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:50.980858 27854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:50.990422 27854 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:41079
I20260806 16:04:50.990449 28035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:41079 every 8 connection(s)
I20260806 16:04:50.991500 27854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:50.993386 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 27854
I20260806 16:04:50.993530 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:50.993875 13904 external_mini_cluster.cc:1428] Setting key 4b2cd0af8d47b5dc06aef3fbcad42f90
I20260806 16:04:51.001454 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:50.993527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:53095 (local address 127.13.148.62:45653)
0806 16:04:50.993725 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:50.993730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:50.994967 (+ 1237us) server_negotiation.cc:407] Connection header received
0806 16:04:50.996560 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:50.996563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:50.996621 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:50.996703 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:50.997651 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.998195 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.999447 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.999681 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.999799 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:51.000935 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:51.000939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:51.000942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:51.000962 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:51.000982 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:51.000990 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:51.001071 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:51.001234 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 16:04:51.001302 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:04:51.002334 28038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:50.993947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45653 (local address 127.13.148.3:53095)
0806 16:04:50.994828 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:50.994886 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:50.996355 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:50.996887 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:50.996894 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:50.996980 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:50.997509 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:50.997523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.998345 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.998348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:50.999310 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:50.999321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:50.999839 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:50.999842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:51.000012 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:51.000624 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:51.000626 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:51.000706 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:51.000776 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:51.001073 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:51.001085 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:51.001092 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:04:51.001233 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 16:04:51.001531 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":158,"threads_started":1}
I20260806 16:04:51.003623 28036 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45653
I20260806 16:04:51.003866 28036 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:51.004372 28036 heartbeater.cc:511] Master 127.13.148.62:45653 requested a full tablet report, sending...
I20260806 16:04:51.005524 27409 ts_manager.cc:194] Registered new tserver with Master: 8f1f75a3b1d94b3596479701b8c0dca8 (127.13.148.3:41079)
I20260806 16:04:51.006153 27409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.3:53095
I20260806 16:04:51.008515 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 16:04:51.012010 27469 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:51.010675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.13.148.62:45653)
0806 16:04:51.010859 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:51.010863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:51.011132 (+ 269us) server_negotiation.cc:407] Connection header received
0806 16:04:51.011221 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:51.011224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:51.011269 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:51.011410 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:51.011858 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49314: BlockingRecv error: recv got EOF from 127.0.0.1:49314 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 16:04:51.012221 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 27500
I20260806 16:04:51.037503 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 27676
I20260806 16:04:51.057900 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 27854
I20260806 16:04:51.073346 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 27369
2026-08-06T16:04:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (972 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T16:04:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:51Z Disabled control of system clock
I20260806 16:04:51.112356 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:32823
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 16:04:51.235169 28049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:51.235483 28049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:51.235523 28049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:51.239742 28049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:51.239813 28049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:51.239830 28049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:51.239847 28049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:51.239863 28049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:51.244848 28049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:32823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:51.245975 28049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:51.247392 28049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:51.256908 28066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:51.256851 28057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:51.256884 28049 server_base.cc:1034] running on GCE node
W20260806 16:04:51.256851 28061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:51.257763 28049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:51.259073 28049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:51.260298 28049 hybrid_clock.cc:648] HybridClock initialized: now 1786032291260277 us; error 34 us; skew 500 ppm
I20260806 16:04:51.262631 28049 webserver.cc:460] Webserver started at http://127.13.148.62:35155/ using document root <none> and password file <none>
I20260806 16:04:51.263275 28049 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:51.263381 28049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:51.263634 28049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:51.265897 28049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "05abc9e4c5a14eaca7fdd6ecdd502797"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "612d34a4da213d67c7c816e5406e3b44"
server_key_iv: "df1b492b09e2f8627752fa5056bc2c73"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.266582 28049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "05abc9e4c5a14eaca7fdd6ecdd502797"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "612d34a4da213d67c7c816e5406e3b44"
server_key_iv: "df1b492b09e2f8627752fa5056bc2c73"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.270532 28049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:04:51.273171 28073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.274684 28049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:51.274845 28049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "05abc9e4c5a14eaca7fdd6ecdd502797"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "612d34a4da213d67c7c816e5406e3b44"
server_key_iv: "df1b492b09e2f8627752fa5056bc2c73"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.274982 28049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:51.279968 28049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:51.280633 28049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:51.280839 28049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:51.289177 28150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:32823 every 8 connection(s)
I20260806 16:04:51.289172 28049 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:32823
I20260806 16:04:51.290334 28049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:51.293951 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:51.299923 28152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797: Bootstrap starting.
I20260806 16:04:51.300184 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 28049
I20260806 16:04:51.300345 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:51.300683 13904 external_mini_cluster.cc:1428] Setting key 4b071e8ef00b174dede23ccf6a44116e
I20260806 16:04:51.302872 28152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:51.303815 28152 log.cc:824] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:51.306430 28152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797: No bootstrap required, opened a new log
I20260806 16:04:51.307374 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:51.302237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.13.148.62:32823)
0806 16:04:51.303062 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:51.303087 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:51.303137 (+ 50us) server_negotiation.cc:407] Connection header received
0806 16:04:51.304028 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:51.304053 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:51.304481 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:51.304968 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:51.305939 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:51.305944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:51.305955 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:51.305966 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:51.306024 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:51.306042 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:51.306170 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:51.306301 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 16:04:51.306633 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":198,"threads_started":1}
I20260806 16:04:51.310978 28152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abc9e4c5a14eaca7fdd6ecdd502797" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32823 } }
I20260806 16:04:51.311223 28152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:51.311259 28152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05abc9e4c5a14eaca7fdd6ecdd502797, State: Initialized, Role: FOLLOWER
I20260806 16:04:51.311723 28152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abc9e4c5a14eaca7fdd6ecdd502797" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32823 } }
I20260806 16:04:51.311852 28152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:51.311895 28152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:51.311991 28152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:51.313017 28152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abc9e4c5a14eaca7fdd6ecdd502797" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32823 } }
I20260806 16:04:51.313390 28152 leader_election.cc:302] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05abc9e4c5a14eaca7fdd6ecdd502797; no voters:
I20260806 16:04:51.313750 28152 leader_election.cc:288] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:51.313954 28160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:51.314222 28160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [term 1 LEADER]: Becoming Leader. State: Replica: 05abc9e4c5a14eaca7fdd6ecdd502797, State: Running, Role: LEADER
I20260806 16:04:51.314656 28160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abc9e4c5a14eaca7fdd6ecdd502797" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32823 } }
I20260806 16:04:51.315243 28152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:51.316654 28164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05abc9e4c5a14eaca7fdd6ecdd502797. Latest consensus state: current_term: 1 leader_uuid: "05abc9e4c5a14eaca7fdd6ecdd502797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abc9e4c5a14eaca7fdd6ecdd502797" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32823 } } }
I20260806 16:04:51.316812 28164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:51.317175 28163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05abc9e4c5a14eaca7fdd6ecdd502797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05abc9e4c5a14eaca7fdd6ecdd502797" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32823 } } }
I20260806 16:04:51.317288 28163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:51.317263 28173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:51.320523 28173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:51.326298 28173 catalog_manager.cc:1347] Generated new cluster ID: 987f6fc49088421c8618f0e8b8b6e313
I20260806 16:04:51.326395 28173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:51.337877 28173 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:51.338270 28173 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:51.355410 28173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05abc9e4c5a14eaca7fdd6ecdd502797: Generated new TSK 0
I20260806 16:04:51.356463 28173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:04:51.363287 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32823
--builtin_ntp_servers=127.13.148.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 16:04:51.489514 28186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:51.489850 28186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:51.489893 28186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:51.494243 28186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:51.494341 28186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:51.494401 28186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:51.499543 28186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:51.500707 28186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:51.502242 28186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:51.512702 28193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:51.512826 28192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:51.512800 28195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:51.513394 28186 server_base.cc:1034] running on GCE node
I20260806 16:04:51.513878 28186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:51.514744 28186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:51.515985 28186 hybrid_clock.cc:648] HybridClock initialized: now 1786032291515963 us; error 35 us; skew 500 ppm
I20260806 16:04:51.518185 28186 webserver.cc:460] Webserver started at http://127.13.148.1:34007/ using document root <none> and password file <none>
I20260806 16:04:51.518874 28186 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:51.518977 28186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:51.519229 28186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:51.521391 28186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "3d8c0ef80092409b9dbb1c0de1d1e255"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "76bdfc598f8b61fdd5a4e5ac818d77ba"
server_key_iv: "472da2787a4e9db8c7c4f08063faf486"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.522008 28186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "3d8c0ef80092409b9dbb1c0de1d1e255"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "76bdfc598f8b61fdd5a4e5ac818d77ba"
server_key_iv: "472da2787a4e9db8c7c4f08063faf486"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.526371 28186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:04:51.529399 28204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.530817 28186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:51.530978 28186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "3d8c0ef80092409b9dbb1c0de1d1e255"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "76bdfc598f8b61fdd5a4e5ac818d77ba"
server_key_iv: "472da2787a4e9db8c7c4f08063faf486"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.531116 28186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:51.536094 28186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:51.536758 28186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:51.536960 28186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:51.537559 28186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:51.538558 28186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:51.538635 28186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.538698 28186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:51.538755 28186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.548324 28186 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:43333
I20260806 16:04:51.548347 28366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:43333 every 8 connection(s)
I20260806 16:04:51.549353 28186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:51.552084 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 28186
I20260806 16:04:51.552240 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:51.552644 13904 external_mini_cluster.cc:1428] Setting key 5c97d673a5a14bd7ff8ecf86aba75d90
I20260806 16:04:51.555152 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32823
--builtin_ntp_servers=127.13.148.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 16:04:51.557518 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:51.551639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:34381 (local address 127.13.148.62:32823)
0806 16:04:51.551903 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:51.551909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:51.553296 (+ 1387us) server_negotiation.cc:407] Connection header received
0806 16:04:51.555187 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:51.555190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:51.555276 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:51.555380 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:51.556944 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:51.556947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:51.556949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:51.556961 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:51.556983 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:51.556992 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:51.557142 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:51.557325 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 16:04:51.557377 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260806 16:04:51.558619 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:51.552097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32823 (local address 127.13.148.1:34381)
0806 16:04:51.553123 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:51.553173 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:51.554877 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:51.555693 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:51.555706 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:51.555846 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:51.556563 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:51.556568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:51.556634 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:51.556747 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:51.557126 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:51.557148 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:51.557157 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 16:04:51.557365 (+ 208us) client_negotiation.cc:240] Negotiation successful
0806 16:04:51.557719 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":173,"threads_started":1}
I20260806 16:04:51.560190 28367 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32823
I20260806 16:04:51.560508 28367 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:51.560989 28367 heartbeater.cc:511] Master 127.13.148.62:32823 requested a full tablet report, sending...
I20260806 16:04:51.562870 28095 ts_manager.cc:194] Registered new tserver with Master: 3d8c0ef80092409b9dbb1c0de1d1e255 (127.13.148.1:43333)
W20260806 16:04:51.685341 28371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:51.685631 28371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:51.685672 28371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:51.689829 28371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:51.689894 28371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:51.689949 28371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:51.694816 28371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:51.695961 28371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:51.697392 28371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:51.707497 28380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:51.707521 28381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:51.707521 28383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:51.707974 28371 server_base.cc:1034] running on GCE node
I20260806 16:04:51.708499 28371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:51.709251 28371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:51.710690 28371 hybrid_clock.cc:648] HybridClock initialized: now 1786032291710443 us; error 42 us; skew 500 ppm
I20260806 16:04:51.712774 28371 webserver.cc:460] Webserver started at http://127.13.148.2:43411/ using document root <none> and password file <none>
I20260806 16:04:51.713438 28371 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:51.713565 28371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:51.713824 28371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:51.716179 28371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "7c35ee29d1074eeb9f97487af79a9e3a"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "f7eb0a72d402fc5abc43e4243d0285a1"
server_key_iv: "586959989c9eac1a33401369799a68cf"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.716913 28371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "7c35ee29d1074eeb9f97487af79a9e3a"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "f7eb0a72d402fc5abc43e4243d0285a1"
server_key_iv: "586959989c9eac1a33401369799a68cf"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.720978 28371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:04:51.723796 28392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.725284 28371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:04:51.725402 28371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "7c35ee29d1074eeb9f97487af79a9e3a"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "f7eb0a72d402fc5abc43e4243d0285a1"
server_key_iv: "586959989c9eac1a33401369799a68cf"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.725569 28371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:51.730582 28371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:51.731257 28371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:51.731477 28371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:51.732084 28371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:51.733083 28371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:51.733134 28371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.733175 28371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:51.733234 28371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.742700 28371 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:42073
I20260806 16:04:51.742731 28552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:42073 every 8 connection(s)
I20260806 16:04:51.743665 28371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:51.744362 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 28371
I20260806 16:04:51.744503 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:51.744798 13904 external_mini_cluster.cc:1428] Setting key ddc12058fe28d6709669ce0e1728af8b
I20260806 16:04:51.747306 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32823
--builtin_ntp_servers=127.13.148.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 16:04:51.751471 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:51.745821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:58691 (local address 127.13.148.62:32823)
0806 16:04:51.745979 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:51.745984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:51.747489 (+ 1505us) server_negotiation.cc:407] Connection header received
0806 16:04:51.749511 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:51.749516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:51.749571 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:51.749692 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:51.750910 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:51.750916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:51.750922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:51.750941 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:51.750969 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:51.750983 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:51.751090 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:51.751217 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 16:04:51.751272 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:04:51.752339 28557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:51.746346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32823 (local address 127.13.148.2:58691)
0806 16:04:51.747250 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:51.747348 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:51.749203 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:51.749840 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:51.749846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:51.749942 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:51.750618 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:51.750620 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:51.750665 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:51.750740 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:51.751077 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:51.751084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:51.751088 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:04:51.751229 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 16:04:51.751455 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":174,"threads_started":1}
I20260806 16:04:51.753643 28553 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32823
I20260806 16:04:51.753897 28553 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:51.754359 28553 heartbeater.cc:511] Master 127.13.148.62:32823 requested a full tablet report, sending...
I20260806 16:04:51.755571 28095 ts_manager.cc:194] Registered new tserver with Master: 7c35ee29d1074eeb9f97487af79a9e3a (127.13.148.2:42073)
W20260806 16:04:51.867218 28558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:51.867535 28558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:51.867578 28558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:51.871847 28558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:51.871924 28558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:51.871985 28558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:51.877343 28558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:51.878731 28558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:51.880342 28558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:51.890914 28566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:51.891074 28558 server_base.cc:1034] running on GCE node
W20260806 16:04:51.891005 28564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:51.891032 28570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:51.891716 28558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:51.892462 28558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:51.893718 28558 hybrid_clock.cc:648] HybridClock initialized: now 1786032291893675 us; error 56 us; skew 500 ppm
I20260806 16:04:51.896072 28558 webserver.cc:460] Webserver started at http://127.13.148.3:38669/ using document root <none> and password file <none>
I20260806 16:04:51.896780 28558 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:51.896858 28558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:51.897156 28558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:51.899634 28558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "2c33b7228fe74ae9a0181bce0cb2a65e"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "5d47bd07510ba71a227853d553332bff"
server_key_iv: "262c449b26f445a3068faea7698ae205"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.900310 28558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "2c33b7228fe74ae9a0181bce0cb2a65e"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "5d47bd07510ba71a227853d553332bff"
server_key_iv: "262c449b26f445a3068faea7698ae205"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.904826 28558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:04:51.908160 28579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.909690 28558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:04:51.909861 28558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "2c33b7228fe74ae9a0181bce0cb2a65e"
format_stamp: "Formatted at 2026-08-06 16:04:51 on dist-test-slave-7kzw"
server_key: "5d47bd07510ba71a227853d553332bff"
server_key_iv: "262c449b26f445a3068faea7698ae205"
server_key_version: "encryptionkey@0"
I20260806 16:04:51.909998 28558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:51.915005 28558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:51.915745 28558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:51.915987 28558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:51.916594 28558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:51.917672 28558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:51.917747 28558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.917815 28558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:51.917866 28558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:51.928974 28558 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:33597
I20260806 16:04:51.928993 28730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:33597 every 8 connection(s)
I20260806 16:04:51.930238 28558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:51.936403 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 28558
I20260806 16:04:51.936529 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:51.936884 13904 external_mini_cluster.cc:1428] Setting key 776d972d7b218d30085279ff791901d5
I20260806 16:04:51.938201 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:51.932431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58309 (local address 127.13.148.62:32823)
0806 16:04:51.932624 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:51.932629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:51.933750 (+ 1121us) server_negotiation.cc:407] Connection header received
0806 16:04:51.935834 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:51.935837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:51.935880 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:51.936083 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:51.937650 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:51.937653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:51.937655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:04:51.937667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:51.937689 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:04:51.937697 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:51.937775 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:51.937985 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 16:04:51.938038 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 16:04:51.939541 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:51.932872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32823 (local address 127.13.148.3:58309)
0806 16:04:51.933586 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:51.933636 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:51.935574 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:51.936347 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:51.936359 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:51.936483 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:51.937335 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:04:51.937336 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:51.937383 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:04:51.937463 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:51.937797 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:51.937808 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:51.937816 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:04:51.938048 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 16:04:51.938334 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":134,"threads_started":1}
I20260806 16:04:51.940784 28731 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32823
I20260806 16:04:51.941073 28731 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:51.941533 28731 heartbeater.cc:511] Master 127.13.148.62:32823 requested a full tablet report, sending...
I20260806 16:04:51.942776 28095 ts_manager.cc:194] Registered new tserver with Master: 2c33b7228fe74ae9a0181bce0cb2a65e (127.13.148.3:33597)
I20260806 16:04:51.952241 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 16:04:51.955752 28156 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:04:51.954555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.13.148.62:32823)
0806 16:04:51.954746 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:51.954751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:51.954910 (+ 159us) server_negotiation.cc:407] Connection header received
0806 16:04:51.955106 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:51.955110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:51.955166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:51.955284 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:51.955580 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38056: BlockingRecv error: recv got EOF from 127.0.0.1:38056 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 16:04:51.956364 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 28186
I20260806 16:04:51.971637 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 28371
I20260806 16:04:51.990751 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 28558
I20260806 16:04:52.004132 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 28049
2026-08-06T16:04:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (931 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:52 dist-test-slave-7kzw krb5kdc[28743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:52 dist-test-slave-7kzw krb5kdc[28743](info): set up 2 sockets
Aug 06 16:04:52 dist-test-slave-7kzw krb5kdc[28743](info): commencing operation
krb5kdc: starting...
W20260806 16:04:53.236517 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:53 dist-test-slave-7kzw krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:04:53.369204 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45071
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:53.481046 28760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:53.481359 28760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:53.481444 28760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:53.485459 28760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:04:53.485560 28760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:53.485601 28760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:53.485659 28760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:04:53.485683 28760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:04:53.490821 28760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:45071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:53.491844 28760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:53.492871 28760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:53.501452 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:53.501502 28767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:53.501437 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:53.501711 28760 server_base.cc:1034] running on GCE node
I20260806 16:04:53.502383 28760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:53.503544 28760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:53.504767 28760 hybrid_clock.cc:648] HybridClock initialized: now 1786032293504743 us; error 32 us; skew 500 ppm
Aug 06 16:04:53 dist-test-slave-7kzw krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032293, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:53.507782 28760 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:04:53.508679 28760 webserver.cc:460] Webserver started at http://127.13.148.62:43315/ using document root <none> and password file <none>
I20260806 16:04:53.509300 28760 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:53.509397 28760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:53.509636 28760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:53.511857 28760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "5aec692dd8434402b021616520da37b6"
format_stamp: "Formatted at 2026-08-06 16:04:53 on dist-test-slave-7kzw"
server_key: "ef72db7e5ab88a3b5bbfa4147512e776"
server_key_iv: "e9e19f21eb2c67e784fd47bbe288dd0b"
server_key_version: "encryptionkey@0"
I20260806 16:04:53.512449 28760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "5aec692dd8434402b021616520da37b6"
format_stamp: "Formatted at 2026-08-06 16:04:53 on dist-test-slave-7kzw"
server_key: "ef72db7e5ab88a3b5bbfa4147512e776"
server_key_iv: "e9e19f21eb2c67e784fd47bbe288dd0b"
server_key_version: "encryptionkey@0"
I20260806 16:04:53.516425 28760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:04:53.518989 28779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:53.520205 28760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:04:53.520371 28760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "5aec692dd8434402b021616520da37b6"
format_stamp: "Formatted at 2026-08-06 16:04:53 on dist-test-slave-7kzw"
server_key: "ef72db7e5ab88a3b5bbfa4147512e776"
server_key_iv: "e9e19f21eb2c67e784fd47bbe288dd0b"
server_key_version: "encryptionkey@0"
I20260806 16:04:53.520491 28760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:53.532821 28760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:53.536072 28760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:53.536288 28760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:53.543740 28760 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:45071
I20260806 16:04:53.544155 28855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:45071 every 8 connection(s)
I20260806 16:04:53.544865 28760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:04:53.546042 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 28760
I20260806 16:04:53.546222 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:04:53.546591 13904 external_mini_cluster.cc:1428] Setting key c558f1547092a01171958e3e5f38cd5c
I20260806 16:04:53.548277 28857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:53.555181 28857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6: Bootstrap starting.
Aug 06 16:04:53 dist-test-slave-7kzw krb5kdc[28743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:53.557557 28857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:53.558310 28857 log.cc:824] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:53.560688 28857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6: No bootstrap required, opened a new log
I20260806 16:04:53.561578 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:53.548035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.13.148.62:45071)
0806 16:04:53.548480 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:53.548496 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:53.548534 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:04:53.549187 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:53.549206 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:53.549532 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:53.549825 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:53.550840 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:53.551843 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:53.552998 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:53.553453 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:53.553570 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:53.556585 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:53.556590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:53.556604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:53.556635 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:53.558411 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:53.558997 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:53.559002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:53.559007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:53.559157 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:53.559452 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:53.559455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:53.559457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:53.559954 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:53.560168 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:53.560347 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 16:04:53.560641 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":158,"threads_started":1}
I20260806 16:04:53.564421 28857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aec692dd8434402b021616520da37b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45071 } }
I20260806 16:04:53.564639 28857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:53.564694 28857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aec692dd8434402b021616520da37b6, State: Initialized, Role: FOLLOWER
I20260806 16:04:53.565035 28857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aec692dd8434402b021616520da37b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45071 } }
I20260806 16:04:53.565145 28857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:04:53.565188 28857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:04:53.565258 28857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:53.566089 28857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aec692dd8434402b021616520da37b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45071 } }
I20260806 16:04:53.566376 28857 leader_election.cc:302] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aec692dd8434402b021616520da37b6; no voters:
I20260806 16:04:53.566733 28857 leader_election.cc:288] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:04:53.566818 28863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:53.567114 28863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [term 1 LEADER]: Becoming Leader. State: Replica: 5aec692dd8434402b021616520da37b6, State: Running, Role: LEADER
I20260806 16:04:53.567718 28863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aec692dd8434402b021616520da37b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45071 } }
I20260806 16:04:53.568092 28857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:04:53.569348 28865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aec692dd8434402b021616520da37b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aec692dd8434402b021616520da37b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45071 } } }
I20260806 16:04:53.569489 28865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:53.569892 28864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aec692dd8434402b021616520da37b6. Latest consensus state: current_term: 1 leader_uuid: "5aec692dd8434402b021616520da37b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aec692dd8434402b021616520da37b6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45071 } } }
I20260806 16:04:53.570010 28864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:04:53.569886 28875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:04:53.572682 28875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:04:53.578174 28875 catalog_manager.cc:1347] Generated new cluster ID: 88c17e70a8044aaa89e3d0366e9e0817
I20260806 16:04:53.578253 28875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:04:53.593166 28875 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:04:53.594588 28875 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:04:53.606392 28875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6: Generated new TSK 0
I20260806 16:04:53.607163 28875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:04:53.645961 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45071
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:53.763785 28888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:53.764053 28888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:53.764096 28888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:53.768154 28888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:53.768211 28888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:53.768296 28888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:04:53.773768 28888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:53.774780 28888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:53.775753 28888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:53.785763 28895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:53.785787 28897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:53.785815 28888 server_base.cc:1034] running on GCE node
W20260806 16:04:53.785822 28894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:53.786590 28888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:53.787189 28888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:53.788411 28888 hybrid_clock.cc:648] HybridClock initialized: now 1786032293788394 us; error 27 us; skew 500 ppm
Aug 06 16:04:53 dist-test-slave-7kzw krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032293, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:53.791504 28888 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:04:53.792356 28888 webserver.cc:460] Webserver started at http://127.13.148.1:37321/ using document root <none> and password file <none>
I20260806 16:04:53.792927 28888 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:53.792995 28888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:53.793166 28888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:53.795289 28888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "1f6ce5021eed47bd90c9136b190f32be"
format_stamp: "Formatted at 2026-08-06 16:04:53 on dist-test-slave-7kzw"
server_key: "f75e9f0e6db8a9a958823c63129e276f"
server_key_iv: "69a0047e5c3b0d7a7f750466fb670d02"
server_key_version: "encryptionkey@0"
I20260806 16:04:53.795835 28888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "1f6ce5021eed47bd90c9136b190f32be"
format_stamp: "Formatted at 2026-08-06 16:04:53 on dist-test-slave-7kzw"
server_key: "f75e9f0e6db8a9a958823c63129e276f"
server_key_iv: "69a0047e5c3b0d7a7f750466fb670d02"
server_key_version: "encryptionkey@0"
I20260806 16:04:53.799602 28888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:04:53.802266 28906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:53.803738 28888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:53.803858 28888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "1f6ce5021eed47bd90c9136b190f32be"
format_stamp: "Formatted at 2026-08-06 16:04:53 on dist-test-slave-7kzw"
server_key: "f75e9f0e6db8a9a958823c63129e276f"
server_key_iv: "69a0047e5c3b0d7a7f750466fb670d02"
server_key_version: "encryptionkey@0"
I20260806 16:04:53.804026 28888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:53.821879 28888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:53.825462 28888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:53.825682 28888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:53.826296 28888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:53.827178 28888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:53.827226 28888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:53.827322 28888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:53.827354 28888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:53.836067 28888 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:36587
I20260806 16:04:53.836088 29060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:36587 every 8 connection(s)
I20260806 16:04:53.836961 28888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:04:53.844537 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 28888
I20260806 16:04:53.844727 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:04:53.845153 13904 external_mini_cluster.cc:1428] Setting key dd74b5244792838372a8164938b40d45
Aug 06 16:04:53 dist-test-slave-7kzw krb5kdc[28743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032293, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:53.852859 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:53.839091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:39747 (local address 127.13.148.62:45071)
0806 16:04:53.839314 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:53.839319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:53.840080 (+ 761us) server_negotiation.cc:407] Connection header received
0806 16:04:53.841677 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:53.841680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:53.841733 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:53.841813 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:53.843293 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:53.843868 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:53.845375 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:53.845694 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:53.845789 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:53.848777 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:53.848785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:53.848791 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:53.848829 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:53.850399 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:53.851020 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:53.851026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:53.851031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:53.851134 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:53.851537 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:53.851543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:53.851548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:53.851820 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:53.851929 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:53.852492 (+ 563us) server_negotiation.cc:299] Negotiation successful
0806 16:04:53.852652 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260806 16:04:53.853613 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:53.839416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45071 (local address 127.13.148.1:39747)
0806 16:04:53.839905 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:53.839952 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:53.841403 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:53.842044 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:53.842052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:53.842519 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:53.843113 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:53.843129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:53.844010 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:53.844013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:53.845207 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:53.845218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:53.845877 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:53.845881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:53.846102 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:53.846871 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:53.846898 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:53.848587 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:53.850574 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:53.850580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:53.850594 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:53.850906 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:53.851280 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:53.851283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:53.851284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:53.851438 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:53.851926 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:53.851931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:53.852176 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 16:04:53.852407 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 16:04:53.852706 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
I20260806 16:04:53.855001 29061 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45071
I20260806 16:04:53.855285 29061 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:53.855803 29061 heartbeater.cc:511] Master 127.13.148.62:45071 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:04:53.857470 28806 ts_manager.cc:194] Registered new tserver with Master: 1f6ce5021eed47bd90c9136b190f32be (127.13.148.1:36587)
I20260806 16:04:53.858793 28806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:39747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:04:53.877491 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45071
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:53.995299 29069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:53.995573 29069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:53.995618 29069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:53.999594 29069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:53.999662 29069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:53.999720 29069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:04:54.004953 29069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:54.005977 29069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:54.006992 29069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:54.017131 29075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:54.017130 29076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:54.017320 29069 server_base.cc:1034] running on GCE node
W20260806 16:04:54.017442 29079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:54.017875 29069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:54.018693 29069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:54.019897 29069 hybrid_clock.cc:648] HybridClock initialized: now 1786032294019880 us; error 27 us; skew 500 ppm
Aug 06 16:04:54 dist-test-slave-7kzw krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032294, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:54.023049 29069 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:04:54.023967 29069 webserver.cc:460] Webserver started at http://127.13.148.2:35367/ using document root <none> and password file <none>
I20260806 16:04:54.024693 29069 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:54.024801 29069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:54.025153 29069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:54.027529 29069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "3c6f6e06918b45a19ea150cca0f32c73"
format_stamp: "Formatted at 2026-08-06 16:04:54 on dist-test-slave-7kzw"
server_key: "321dfe13ac45113ee4429bf017df3dd0"
server_key_iv: "41599d608058ee5d6956dd70fe111b3e"
server_key_version: "encryptionkey@0"
I20260806 16:04:54.028139 29069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "3c6f6e06918b45a19ea150cca0f32c73"
format_stamp: "Formatted at 2026-08-06 16:04:54 on dist-test-slave-7kzw"
server_key: "321dfe13ac45113ee4429bf017df3dd0"
server_key_iv: "41599d608058ee5d6956dd70fe111b3e"
server_key_version: "encryptionkey@0"
I20260806 16:04:54.032119 29069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:04:54.035246 29088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:54.036593 29069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:04:54.036779 29069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "3c6f6e06918b45a19ea150cca0f32c73"
format_stamp: "Formatted at 2026-08-06 16:04:54 on dist-test-slave-7kzw"
server_key: "321dfe13ac45113ee4429bf017df3dd0"
server_key_iv: "41599d608058ee5d6956dd70fe111b3e"
server_key_version: "encryptionkey@0"
I20260806 16:04:54.036911 29069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:54.047346 29069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:54.050709 29069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:54.050918 29069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:54.051501 29069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:54.052314 29069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:54.052386 29069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:54.052459 29069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:54.052501 29069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:54.061494 29069 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:41315
I20260806 16:04:54.061518 29263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:41315 every 8 connection(s)
I20260806 16:04:54.062381 29069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:04:54.065855 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 29069
I20260806 16:04:54.065973 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:04:54.066270 13904 external_mini_cluster.cc:1428] Setting key 1837d439866f3b14ce68b1da3df517fa
Aug 06 16:04:54 dist-test-slave-7kzw krb5kdc[28743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032294, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:04:54.078981 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.064395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:58585 (local address 127.13.148.62:45071)
0806 16:04:54.064687 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.064696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.065667 (+ 971us) server_negotiation.cc:407] Connection header received
0806 16:04:54.067640 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.067643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.067732 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.067827 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:54.069355 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.070221 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.071567 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.071800 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.071912 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.074926 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:54.074934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:54.074938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:54.074977 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:54.076750 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:54.077280 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:54.077286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:54.077291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:54.077393 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:54.077762 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:54.077768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:54.077773 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:54.078037 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:54.078151 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.078683 (+ 532us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.078793 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:04:54.079833 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.064802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45071 (local address 127.13.148.2:58585)
0806 16:04:54.065446 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:54.065517 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:54.067409 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:54.067996 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:54.068003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:54.068490 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:54.069179 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.069194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.070372 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.070377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.071423 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.071435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.071952 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.071956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.072157 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.072820 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:54.072836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:54.074744 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:54.076875 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:54.076880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:54.076892 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:54.077151 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:54.077495 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:54.077498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:54.077499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:54.077633 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:54.078146 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:54.078153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:54.078389 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 16:04:54.078602 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 16:04:54.078934 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260806 16:04:54.081185 29265 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45071
I20260806 16:04:54.081451 29265 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:54.082101 29265 heartbeater.cc:511] Master 127.13.148.62:45071 requested a full tablet report, sending...
I20260806 16:04:54.083357 28806 ts_manager.cc:194] Registered new tserver with Master: 3c6f6e06918b45a19ea150cca0f32c73 (127.13.148.2:41315)
I20260806 16:04:54.083936 28806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:58585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:04:54.097532 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45071
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:04:54.213574 29272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:04:54.213843 29272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:04:54.213887 29272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:04:54.217957 29272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:04:54.218019 29272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:04:54.218115 29272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:04:54.223464 29272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:04:54.224568 29272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:04:54.225634 29272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:04:54.235546 29280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:54.235672 29272 server_base.cc:1034] running on GCE node
W20260806 16:04:54.235515 29279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:04:54.235514 29283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:04:54.236239 29272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:04:54.236869 29272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:04:54.238065 29272 hybrid_clock.cc:648] HybridClock initialized: now 1786032294238048 us; error 40 us; skew 500 ppm
Aug 06 16:04:54 dist-test-slave-7kzw krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032294, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:04:54.241000 29272 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:04:54.241907 29272 webserver.cc:460] Webserver started at http://127.13.148.3:43867/ using document root <none> and password file <none>
I20260806 16:04:54.242530 29272 fs_manager.cc:359] Metadata directory not provided
I20260806 16:04:54.242637 29272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:04:54.242870 29272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:04:54.244984 29272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "30404a5b0fbc41f1ba17079932726e70"
format_stamp: "Formatted at 2026-08-06 16:04:54 on dist-test-slave-7kzw"
server_key: "02fc49b1a66a2d364da96bf578b9a954"
server_key_iv: "fa5f147a088cbdf6bb65942786271ad7"
server_key_version: "encryptionkey@0"
I20260806 16:04:54.245609 29272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "30404a5b0fbc41f1ba17079932726e70"
format_stamp: "Formatted at 2026-08-06 16:04:54 on dist-test-slave-7kzw"
server_key: "02fc49b1a66a2d364da96bf578b9a954"
server_key_iv: "fa5f147a088cbdf6bb65942786271ad7"
server_key_version: "encryptionkey@0"
I20260806 16:04:54.249475 29272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:04:54.252103 29292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:54.253276 29272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:04:54.253412 29272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "30404a5b0fbc41f1ba17079932726e70"
format_stamp: "Formatted at 2026-08-06 16:04:54 on dist-test-slave-7kzw"
server_key: "02fc49b1a66a2d364da96bf578b9a954"
server_key_iv: "fa5f147a088cbdf6bb65942786271ad7"
server_key_version: "encryptionkey@0"
I20260806 16:04:54.253523 29272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:04:54.270807 29272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:04:54.273769 29272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:04:54.273972 29272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:04:54.274610 29272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:04:54.275452 29272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:04:54.275525 29272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:54.275600 29272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:04:54.275645 29272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:04:54.284595 29272 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:41457
I20260806 16:04:54.284637 29442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:41457 every 8 connection(s)
I20260806 16:04:54.285465 29272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:04:54.295492 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 29272
I20260806 16:04:54.295643 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:04:54.295975 13904 external_mini_cluster.cc:1428] Setting key 28d6639b8c40071c678341df5293837e
Aug 06 16:04:54 dist-test-slave-7kzw krb5kdc[28743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032294, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:04:54.300741 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.287407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58815 (local address 127.13.148.62:45071)
0806 16:04:54.287593 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.287600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.288399 (+ 799us) server_negotiation.cc:407] Connection header received
0806 16:04:54.290020 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.290025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.290092 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.290228 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:54.291610 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.292232 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.293568 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.293826 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.293970 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.296858 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:54.296865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:54.296868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:54.296898 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:54.298834 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:54.299313 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:54.299316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:54.299317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:54.299398 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:54.299715 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:54.299718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:54.299719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:54.299906 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:54.299989 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.300464 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.300578 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 16:04:54.301568 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.287741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45071 (local address 127.13.148.3:58815)
0806 16:04:54.288228 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:54.288279 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:54.289769 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:54.290420 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:54.290428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:54.290902 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:04:54.291421 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.291437 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.292409 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.292414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.293402 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.293413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.294013 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.294018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.294230 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.294849 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:04:54.294866 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:04:54.296693 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:04:54.298941 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:54.298946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:54.298959 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:54.299222 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:54.299479 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:04:54.299481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:04:54.299482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:04:54.299614 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:04:54.299998 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:04:54.300002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:04:54.300249 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 16:04:54.300406 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 16:04:54.300687 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":117,"threads_started":1}
I20260806 16:04:54.302788 29443 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45071
I20260806 16:04:54.303032 29443 heartbeater.cc:464] Registering TS with master...
I20260806 16:04:54.303516 29443 heartbeater.cc:511] Master 127.13.148.62:45071 requested a full tablet report, sending...
I20260806 16:04:54.304575 28806 ts_manager.cc:194] Registered new tserver with Master: 30404a5b0fbc41f1ba17079932726e70 (127.13.148.3:41457)
I20260806 16:04:54.305154 28806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:58815
I20260806 16:04:54.309955 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:04:54.319329 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.311967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.13.148.62:45071)
0806 16:04:54.312087 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.312090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.312240 (+ 150us) server_negotiation.cc:407] Connection header received
0806 16:04:54.312334 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.312336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.312373 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.312441 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:04:54.313258 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.313753 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.314582 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.314831 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.314914 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.316216 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:04:54.316219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:04:54.316221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:04:54.316243 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:04:54.317770 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:54.318218 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:54.318220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:54.318222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:54.318299 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:04:54.318603 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:04:54.318606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:04:54.318607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:04:54.318826 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:04:54.318934 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.319117 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.319208 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:04:54.322408 28806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:04:54.325759 28806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:04:54.340938 29457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.334409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:36587 (local address 127.0.0.1:56456)
0806 16:04:54.334821 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:54.334876 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:54.335009 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:54.336321 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:54.336328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:54.336353 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:54.336544 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.336554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.338311 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.338315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.339454 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.339463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.340357 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.340361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.340471 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.340513 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 16:04:54.340637 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 16:04:54.340765 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260806 16:04:54.340999 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.334670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:41457 (local address 127.0.0.1:52494)
0806 16:04:54.335408 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:54.335425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:54.335524 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:54.336537 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:54.336540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:54.336559 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:54.336689 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.336700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.338426 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.338429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.339360 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.339370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.340551 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.340555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.340671 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.340697 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 16:04:54.340757 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 16:04:54.340849 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":101,"threads_started":1}
I20260806 16:04:54.341441 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.335205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41315 (local address 127.0.0.1:43766)
0806 16:04:54.335759 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:54.335773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:54.335863 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:54.336736 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:54.336742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:54.336763 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:54.336862 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.336866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.337887 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.337892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.339423 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.339435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.340618 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.340625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.340724 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.340748 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:04:54.340820 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 16:04:54.340880 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260806 16:04:54.341864 29462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.335238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.13.148.3:41457)
0806 16:04:54.336111 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.336124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.336145 (+ 21us) server_negotiation.cc:407] Connection header received
0806 16:04:54.336221 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.336229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.336403 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.336547 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:54.336824 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.338258 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.339518 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.340406 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.341409 (+ 1003us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.341558 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.341608 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.341711 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":178,"threads_started":1}
I20260806 16:04:54.341931 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.335628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.13.148.2:41315)
0806 16:04:54.336048 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.336056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.336075 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:04:54.336135 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.336139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.336317 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.336562 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:54.336992 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.337752 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.339568 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.340443 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.340546 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.340685 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.340874 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.340971 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":155,"threads_started":1}
I20260806 16:04:54.341958 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.334601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.13.148.1:36587)
0806 16:04:54.335771 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.335789 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.335819 (+ 30us) server_negotiation.cc:407] Connection header received
0806 16:04:54.335902 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.335910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.336146 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.336310 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:54.336693 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.337960 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.339597 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.340208 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.340334 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.340496 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.341029 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.341788 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":93,"threads_started":1}
I20260806 16:04:54.344519 29352 tablet_service.cc:1467] Processing CreateTablet for tablet 50d9ecf7674146ad903aef1b6645317a (DEFAULT_TABLE table=test-table [id=93a5dea7e2864c1e87125be7f638f20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:54.344661 28973 tablet_service.cc:1467] Processing CreateTablet for tablet 50d9ecf7674146ad903aef1b6645317a (DEFAULT_TABLE table=test-table [id=93a5dea7e2864c1e87125be7f638f20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:54.344692 29161 tablet_service.cc:1467] Processing CreateTablet for tablet 50d9ecf7674146ad903aef1b6645317a (DEFAULT_TABLE table=test-table [id=93a5dea7e2864c1e87125be7f638f20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:04:54.345634 29352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d9ecf7674146ad903aef1b6645317a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:54.345816 29161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d9ecf7674146ad903aef1b6645317a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:54.345816 28973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d9ecf7674146ad903aef1b6645317a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:04:54.352051 29464 tablet_bootstrap.cc:492] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: Bootstrap starting.
I20260806 16:04:54.353598 29468 tablet_bootstrap.cc:492] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be: Bootstrap starting.
I20260806 16:04:54.354715 29464 tablet_bootstrap.cc:654] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:54.354801 29466 tablet_bootstrap.cc:492] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: Bootstrap starting.
I20260806 16:04:54.355501 29464 log.cc:824] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:54.355710 29468 tablet_bootstrap.cc:654] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:54.356505 29468 log.cc:824] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:54.357334 29466 tablet_bootstrap.cc:654] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: Neither blocks nor log segments found. Creating new log.
I20260806 16:04:54.358016 29464 tablet_bootstrap.cc:492] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: No bootstrap required, opened a new log
I20260806 16:04:54.358114 29466 log.cc:824] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: Log is configured to *not* fsync() on all Append() calls
I20260806 16:04:54.358251 29464 ts_tablet_manager.cc:1397] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 16:04:54.358556 29468 tablet_bootstrap.cc:492] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be: No bootstrap required, opened a new log
I20260806 16:04:54.358773 29468 ts_tablet_manager.cc:1397] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260806 16:04:54.360107 29466 tablet_bootstrap.cc:492] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: No bootstrap required, opened a new log
I20260806 16:04:54.360317 29466 ts_tablet_manager.cc:1397] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:04:54.361397 29464 raft_consensus.cc:348] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.361644 29464 raft_consensus.cc:374] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:54.361702 29464 raft_consensus.cc:729] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30404a5b0fbc41f1ba17079932726e70, State: Initialized, Role: FOLLOWER
I20260806 16:04:54.362181 29464 consensus_queue.cc:260] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.363008 29468 raft_consensus.cc:348] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.363246 29464 ts_tablet_manager.cc:1428] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260806 16:04:54.363260 29468 raft_consensus.cc:374] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:54.363310 29468 raft_consensus.cc:729] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6ce5021eed47bd90c9136b190f32be, State: Initialized, Role: FOLLOWER
I20260806 16:04:54.363616 29443 heartbeater.cc:503] Master 127.13.148.62:45071 was elected leader, sending a full tablet report...
I20260806 16:04:54.364040 29466 raft_consensus.cc:348] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.364285 29466 raft_consensus.cc:374] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:04:54.364343 29466 raft_consensus.cc:729] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6f6e06918b45a19ea150cca0f32c73, State: Initialized, Role: FOLLOWER
I20260806 16:04:54.364780 29466 consensus_queue.cc:260] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.364815 29468 consensus_queue.cc:260] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.365638 29265 heartbeater.cc:503] Master 127.13.148.62:45071 was elected leader, sending a full tablet report...
I20260806 16:04:54.365728 29468 ts_tablet_manager.cc:1428] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:04:54.365728 29466 ts_tablet_manager.cc:1428] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 16:04:54.366717 29061 heartbeater.cc:503] Master 127.13.148.62:45071 was elected leader, sending a full tablet report...
I20260806 16:04:54.443257 29473 raft_consensus.cc:482] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:04:54.443526 29473 raft_consensus.cc:504] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.444803 29473 leader_election.cc:288] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f6ce5021eed47bd90c9136b190f32be (127.13.148.1:36587), 30404a5b0fbc41f1ba17079932726e70 (127.13.148.3:41457)
I20260806 16:04:54.449347 29462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.445281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:48571 (local address 127.13.148.3:41457)
0806 16:04:54.445435 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.445441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.445461 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:04:54.445724 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.445750 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.445824 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.445926 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:54.446310 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.446966 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.448242 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.448696 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.448831 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.448894 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.449115 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.449200 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:04:54.449354 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.445146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:41457 (local address 127.13.148.2:48571)
0806 16:04:54.445387 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:54.445405 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:54.445514 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:54.445949 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:54.445951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:54.445991 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:54.446150 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.446160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.447094 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.447100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.448090 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.448101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.448830 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.448835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.448957 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.448985 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 16:04:54.449061 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 16:04:54.449158 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":38,"spinlock_wait_cycles":16128}
I20260806 16:04:54.449399 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.445270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39821 (local address 127.13.148.1:36587)
0806 16:04:54.445406 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.445412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.445776 (+ 364us) server_negotiation.cc:407] Connection header received
0806 16:04:54.445842 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.445847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.445913 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.445994 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:04:54.446357 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.446986 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.448278 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.448719 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.448870 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.448932 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.449184 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.449280 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:04:54.449376 29477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.445146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:36587 (local address 127.13.148.2:39821)
0806 16:04:54.445645 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:54.445661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:04:54.445770 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:04:54.446011 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:04:54.446014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:04:54.446052 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:04:54.446188 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.446198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.447157 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.447160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.448101 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:04:54.448112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.448888 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.448892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:04:54.449001 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.449020 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:04:54.449137 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 16:04:54.449209 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":84,"spinlock_wait_cycles":3840,"thread_start_us":157,"threads_started":1}
I20260806 16:04:54.450169 28998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d9ecf7674146ad903aef1b6645317a" candidate_uuid: "3c6f6e06918b45a19ea150cca0f32c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6ce5021eed47bd90c9136b190f32be" is_pre_election: true
I20260806 16:04:54.450170 29381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d9ecf7674146ad903aef1b6645317a" candidate_uuid: "3c6f6e06918b45a19ea150cca0f32c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30404a5b0fbc41f1ba17079932726e70" is_pre_election: true
I20260806 16:04:54.450524 28998 raft_consensus.cc:2454] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6f6e06918b45a19ea150cca0f32c73 in term 0.
I20260806 16:04:54.450544 29381 raft_consensus.cc:2454] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6f6e06918b45a19ea150cca0f32c73 in term 0.
I20260806 16:04:54.451067 29090 leader_election.cc:302] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f6ce5021eed47bd90c9136b190f32be, 30404a5b0fbc41f1ba17079932726e70, 3c6f6e06918b45a19ea150cca0f32c73; no voters:
I20260806 16:04:54.451401 29473 raft_consensus.cc:2781] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:04:54.451512 29473 raft_consensus.cc:482] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:04:54.451582 29473 raft_consensus.cc:3037] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:54.452775 29473 raft_consensus.cc:504] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.453214 29473 leader_election.cc:288] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [CANDIDATE]: Term 1 election: Requested vote from peers 1f6ce5021eed47bd90c9136b190f32be (127.13.148.1:36587), 30404a5b0fbc41f1ba17079932726e70 (127.13.148.3:41457)
I20260806 16:04:54.453624 29381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d9ecf7674146ad903aef1b6645317a" candidate_uuid: "3c6f6e06918b45a19ea150cca0f32c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30404a5b0fbc41f1ba17079932726e70"
I20260806 16:04:54.453699 28998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d9ecf7674146ad903aef1b6645317a" candidate_uuid: "3c6f6e06918b45a19ea150cca0f32c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6ce5021eed47bd90c9136b190f32be"
I20260806 16:04:54.453778 29381 raft_consensus.cc:3037] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:54.453861 28998 raft_consensus.cc:3037] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:04:54.455408 29381 raft_consensus.cc:2454] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6f6e06918b45a19ea150cca0f32c73 in term 1.
I20260806 16:04:54.455413 28998 raft_consensus.cc:2454] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6f6e06918b45a19ea150cca0f32c73 in term 1.
I20260806 16:04:54.455775 29089 leader_election.cc:302] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30404a5b0fbc41f1ba17079932726e70, 3c6f6e06918b45a19ea150cca0f32c73; no voters:
I20260806 16:04:54.455964 29473 raft_consensus.cc:2781] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:04:54.456220 29473 raft_consensus.cc:686] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6f6e06918b45a19ea150cca0f32c73, State: Running, Role: LEADER
I20260806 16:04:54.456589 29473 consensus_queue.cc:237] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } }
I20260806 16:04:54.459956 28804 catalog_manager.cc:5495] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c6f6e06918b45a19ea150cca0f32c73 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "3c6f6e06918b45a19ea150cca0f32c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c6f6e06918b45a19ea150cca0f32c73" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 } health_report { overall_health: UNKNOWN } } }
I20260806 16:04:54.494200 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.490459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.13.148.2:41315)
0806 16:04:54.490628 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 16:04:54.490633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:04:54.490650 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:04:54.490814 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:04:54.490817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:04:54.490878 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:04:54.490998 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:04:54.491257 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.491769 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.492648 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:04:54.492877 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:04:54.492996 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:04:54.493327 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:04:54.493767 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:04:54.493906 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 16:04:54.493973 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 16:04:54.494036 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:04:54.500599 29381 raft_consensus.cc:1261] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 1 FOLLOWER]: Refusing update from remote peer 3c6f6e06918b45a19ea150cca0f32c73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:54.500620 28998 raft_consensus.cc:1261] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 1 FOLLOWER]: Refusing update from remote peer 3c6f6e06918b45a19ea150cca0f32c73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:04:54.501435 29478 consensus_queue.cc:1035] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30404a5b0fbc41f1ba17079932726e70" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:54.501718 29473 consensus_queue.cc:1035] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6ce5021eed47bd90c9136b190f32be" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:04:54.510533 29484 mvcc.cc:205] Tried to move back new op lower bound from 7315588278266413056 to 7315588278097317888. Current Snapshot: MvccSnapshot[applied={T|T < 7315588278266413056}]
I20260806 16:04:54.519799 28804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:54.520002 28804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:04:54.522815 28804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5aec692dd8434402b021616520da37b6: Sending DeleteTablet for 3 replicas of tablet 50d9ecf7674146ad903aef1b6645317a
I20260806 16:04:54.523741 28973 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d9ecf7674146ad903aef1b6645317a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:54 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:56456
I20260806 16:04:54.523773 29161 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d9ecf7674146ad903aef1b6645317a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:54 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43766
I20260806 16:04:54.524327 29495 tablet_replica.cc:330] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be: stopping tablet replica
I20260806 16:04:54.524341 29352 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d9ecf7674146ad903aef1b6645317a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:04:54 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52494
I20260806 16:04:54.524873 29495 raft_consensus.cc:2229] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:54.524930 29496 tablet_replica.cc:330] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: stopping tablet replica
I20260806 16:04:54.525224 29496 raft_consensus.cc:2229] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:04:54.525368 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 28888
I20260806 16:04:54.525408 29495 raft_consensus.cc:2258] T 50d9ecf7674146ad903aef1b6645317a P 1f6ce5021eed47bd90c9136b190f32be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:54.525619 29496 raft_consensus.cc:2258] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:54.528949 29496 ts_tablet_manager.cc:1905] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:54.532061 29496 ts_tablet_manager.cc:1918] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:54.532186 29496 log.cc:1196] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/50d9ecf7674146ad903aef1b6645317a
I20260806 16:04:54.532318 29497 tablet_replica.cc:330] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: stopping tablet replica
I20260806 16:04:54.532624 29496 ts_tablet_manager.cc:1939] T 50d9ecf7674146ad903aef1b6645317a P 3c6f6e06918b45a19ea150cca0f32c73: Deleting consensus metadata
I20260806 16:04:54.532846 29497 raft_consensus.cc:2229] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:04:54.533274 29497 raft_consensus.cc:2258] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:04:54.534034 28783 catalog_manager.cc:4841] TS 3c6f6e06918b45a19ea150cca0f32c73 (127.13.148.2:41315): tablet 50d9ecf7674146ad903aef1b6645317a (table test-table [id=93a5dea7e2864c1e87125be7f638f20c]) successfully deleted
I20260806 16:04:54.535300 29497 ts_tablet_manager.cc:1905] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:04:54.538968 29497 ts_tablet_manager.cc:1918] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:04:54.539098 29497 log.cc:1196] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/50d9ecf7674146ad903aef1b6645317a
W20260806 16:04:54.539147 28783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:36587 (error 108)
I20260806 16:04:54.539182 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 29069
I20260806 16:04:54.541759 29497 ts_tablet_manager.cc:1939] T 50d9ecf7674146ad903aef1b6645317a P 30404a5b0fbc41f1ba17079932726e70: Deleting consensus metadata
I20260806 16:04:54.543498 28780 catalog_manager.cc:4841] TS 30404a5b0fbc41f1ba17079932726e70 (127.13.148.3:41457): tablet 50d9ecf7674146ad903aef1b6645317a (table test-table [id=93a5dea7e2864c1e87125be7f638f20c]) successfully deleted
I20260806 16:04:54.546186 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:04:54.545718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:36587 (local address 127.0.0.1:56470)
0806 16:04:54.545989 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 16:04:54.546083 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:36587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260806 16:04:54.546365 28783 catalog_manager.cc:4584] TS 1f6ce5021eed47bd90c9136b190f32be (127.13.148.1:36587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50d9ecf7674146ad903aef1b6645317a: Network error: Client connection negotiation failed: client connection to 127.13.148.1:36587: connect: Connection refused (error 111)
I20260806 16:04:54.555018 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 29272
I20260806 16:04:54.572965 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 28760
2026-08-06T16:04:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2567 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:04:54 dist-test-slave-7kzw krb5kdc[29502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:04:54 dist-test-slave-7kzw krb5kdc[29502](info): set up 2 sockets
Aug 06 16:04:54 dist-test-slave-7kzw krb5kdc[29502](info): commencing operation
krb5kdc: starting...
W20260806 16:04:55.805406 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:04:55 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:04:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:04:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/rangeradmin_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/rangeradmin_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/rangerlookup_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/rangerlookup_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
I20260806 16:04:55.995410 13904 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwTlxSD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/postgres -l logfile start
2026-08-06 16:04:58.657 UTC [29539] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 16:04:58.657 UTC [29539] LOG: listening on IPv4 address "127.13.148.20", port 54199
2026-08-06 16:04:58.661 UTC [29539] LOG: listening on Unix socket "/tmp/.s.PGSQL.54199"
2026-08-06 16:04:58.686 UTC [29544] LOG: database system was shut down at 2026-08-06 16:04:57 UTC
2026-08-06 16:04:58.691 UTC [29539] LOG: database system is ready to accept connections
I20260806 16:04:59.889328 13904 mini_postgres.cc:96] Postgres bound to 54199
2026-08-06 16:04:59.897 UTC [29555] FATAL: database "slave" does not exist
127.13.148.20:54199 - accepting connections
I20260806 16:04:59.897995 13904 mini_ranger.cc:161] Starting Ranger...
I20260806 16:04:59.914695 13904 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 16:05:00.467878 13904 mini_ranger.cc:87] Created ranger Postgres database
I20260806 16:05:00.468001 13904 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-admin
2026-08-06 16:05:00,787 [I] DB FLAVOR :POSTGRES
2026-08-06 16:05:00,787 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 16:05:00,787 [I] Checking connection..
2026-08-06 16:05:00,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 16:05:01,055 [I] Checking connection passed.
2026-08-06 16:05:01,055 [I] --------- Verifying version history table ---------
2026-08-06 16:05:01,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:05:01,323 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 16:05:01,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:05:01,590 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 16:05:01,590 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 16:05:01,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 16:05:01.867 UTC [29642] WARNING: there is no transaction in progress
2026-08-06 16:05:01,874 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 16:05:01,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:05:02,133 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 16:05:02,134 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 16:05:02,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 16:05:02,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 16:05:02,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7kzw.c.gcp-upstream.internal','N') ;"
2026-08-06 16:05:02,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 16:05:03,167 [I] Table x_portal_user does not exist in database ranger
2026-08-06 16:05:03,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 16:05:03,426 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 16:05:03,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 16:05:03,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 16:05:03,970 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 16:05:03,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 16:05:04.962 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:04.979 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:04.996 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.011 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.027 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.046 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.109 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.117 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.131 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.142 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.153 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.167 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.178 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.185 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.192 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05.198 UTC [29797] WARNING: there is no transaction in progress
2026-08-06 16:05:05,880 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 16:05:05,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 16:05:06,121 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 16:05:06,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 16:05:06,369 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 16:05:06,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 16:05:06,654 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 16:05:06,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 16:05:06,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 16:05:07,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7kzw.c.gcp-upstream.internal';"
2026-08-06 16:05:07,421 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 16:05:07,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 16:05:07,677 [I] DB_PATCHES have already been applied
I20260806 16:05:07.683188 13904 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwTlxSD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 16:05:07.683285 13904 mini_ranger.cc:193] Using host: 127.13.148.20
I20260806 16:05:07.688158 13904 mini_ranger.cc:227] Ranger admin URL: http://127.13.148.20:35363
Aug 06, 2026 4:05:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:05:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:05:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:05:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:05:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:05:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:05:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 16:05:09.020812 13904 mini_ranger.cc:160] Time spent starting Ranger: real 9.123s user 0.007s sys 0.007s
Aug 06, 2026 4:05:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:05:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 4:05:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:05:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:05:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 16:05:30.841921 13904 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/rangerkms_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/rangerkms_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.148.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 16:05:30 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032330, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 16:05:30.942248 13904 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 16:05:30.954566 13904 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 16:05:31.928830 13904 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 16:05:31.928944 13904 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms
I20260806 16:05:31.929052 13904 mini_ranger_kms.cc:227] Using postgres at 127.13.148.20:54199
2026-08-06 16:05:32,271 [I] DB FLAVOR :POSTGRES
2026-08-06 16:05:32,272 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 16:05:32,272 [I] Checking connection
2026-08-06 16:05:32,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 16:05:32,549 [I] connection success
2026-08-06 16:05:32,550 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 16:05:32,550 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 16:05:32,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 16:05:32,803 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 16:05:32,803 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 16:05:32,803 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 16:05:32,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:54199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 16:05:33,092 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 16:05:33.584482 13904 mini_ranger_kms.cc:310] Created kms service
I20260806 16:05:33.765949 13904 mini_ranger_kms.cc:326] Created kudu user
I20260806 16:05:33.811460 13904 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 16:05:33 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032330, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:05:34.241492 13904 mini_ranger_kms.cc:384] Added ranger policy
I20260806 16:05:34.241662 13904 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 16:05:34.241683 13904 mini_ranger_kms.cc:240] Using host: 127.13.148.20
I20260806 16:05:34.247888 13904 mini_ranger_kms.cc:276] Ranger KMS PID: 30039
I20260806 16:05:34.248049 13904 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.148.20:58399
Aug 06, 2026 4:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:05:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:05:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 16:05:36.918426 13904 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.976s user 0.009s sys 0.316s
I20260806 16:05:36.918617 13904 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 16:05:36.918752 13904 mini_ranger_kms.cc:401] 127.13.148.20:58399/kms/v1/keys
Aug 06 16:05:36 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 16:05:37 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Aug 06 16:05:37 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Aug 06 16:05:40 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:05:40.141847 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46269
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46269
--ranger_config_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:58399/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:05:40.263824 30121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:05:40.264124 30121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:05:40.264165 30121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:05:40.268896 30121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:05:40.268968 30121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:05:40.268988 30121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:05:40.269006 30121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:05:40.269022 30121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:05:40.275134 30121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.148.20:58399/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46269
--ranger_config_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:05:40.276248 30121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:05:40.277304 30121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:05:40.286743 30130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:05:40.286758 30133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:05:40.286753 30131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:05:40.286804 30121 server_base.cc:1034] running on GCE node
I20260806 16:05:40.287607 30121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:05:40.288796 30121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:05:40.289984 30121 hybrid_clock.cc:648] HybridClock initialized: now 1786032340289965 us; error 33 us; skew 500 ppm
Aug 06 16:05:40 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032340, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:05:40.297953 30121 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:05:40.299014 30121 webserver.cc:460] Webserver started at http://127.13.148.62:37627/ using document root <none> and password file <none>
I20260806 16:05:40.299602 30121 fs_manager.cc:359] Metadata directory not provided
I20260806 16:05:40.299669 30121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:05:40.299898 30121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:05:40 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032340, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:05:40.366185 30121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "3af43456e72d47398d2fbcf332489f55"
format_stamp: "Formatted at 2026-08-06 16:05:40 on dist-test-slave-7kzw"
server_key: "c75a3b7c0ef0c9cd7af068e1b26eb74c"
server_key_iv: "4860e68e80713c04c3e191c3687f7752"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:40.366915 30121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "3af43456e72d47398d2fbcf332489f55"
format_stamp: "Formatted at 2026-08-06 16:05:40 on dist-test-slave-7kzw"
server_key: "c75a3b7c0ef0c9cd7af068e1b26eb74c"
server_key_iv: "4860e68e80713c04c3e191c3687f7752"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:40.371191 30121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:05:40.386653 30143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:40.388264 30121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:05:40.388425 30121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "3af43456e72d47398d2fbcf332489f55"
format_stamp: "Formatted at 2026-08-06 16:05:40 on dist-test-slave-7kzw"
server_key: "c75a3b7c0ef0c9cd7af068e1b26eb74c"
server_key_iv: "4860e68e80713c04c3e191c3687f7752"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:40.388578 30121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:05:40.417001 30121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:05:40.420181 30121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:05:40.420413 30121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:05:40.429438 30121 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46269
I20260806 16:05:40.429479 30218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46269 every 8 connection(s)
I20260806 16:05:40.430572 30121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:05:40.431614 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 30121
I20260806 16:05:40.431741 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:05:40.432046 13904 external_mini_cluster.cc:1428] Setting key ed70115624dae3e750da42cb98449d66
I20260806 16:05:40.434206 30219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:05:40.441644 30219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55: Bootstrap starting.
Aug 06 16:05:40 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:05:40.444861 30219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55: Neither blocks nor log segments found. Creating new log.
I20260806 16:05:40.445843 30219 log.cc:824] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55: Log is configured to *not* fsync() on all Append() calls
I20260806 16:05:40.448494 30219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55: No bootstrap required, opened a new log
I20260806 16:05:40.450457 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:40.433534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.13.148.62:46269)
0806 16:05:40.434023 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:40.434042 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:40.434089 (+ 47us) server_negotiation.cc:407] Connection header received
0806 16:05:40.435052 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:40.435075 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:40.435514 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:40.435910 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:05:40.436788 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:40.438529 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:40.439771 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:40.440550 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:40.440734 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:40.444087 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:05:40.444094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:05:40.444136 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:05:40.444178 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:05:40.447019 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:40.447614 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:40.447620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:40.447627 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:40.447779 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:40.448144 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:40.448147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:40.448149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:40.448660 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:05:40.448857 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:40.449224 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 16:05:40.449504 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":155,"threads_started":1}
I20260806 16:05:40.452471 30219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af43456e72d47398d2fbcf332489f55" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46269 } }
I20260806 16:05:40.452765 30219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:05:40.452864 30219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af43456e72d47398d2fbcf332489f55, State: Initialized, Role: FOLLOWER
I20260806 16:05:40.453375 30219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af43456e72d47398d2fbcf332489f55" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46269 } }
I20260806 16:05:40.453544 30219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:05:40.453641 30219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:05:40.453785 30219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:05:40.455044 30219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af43456e72d47398d2fbcf332489f55" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46269 } }
I20260806 16:05:40.455442 30219 leader_election.cc:302] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af43456e72d47398d2fbcf332489f55; no voters:
I20260806 16:05:40.455736 30219 leader_election.cc:288] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:05:40.455858 30229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:05:40.456091 30229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [term 1 LEADER]: Becoming Leader. State: Replica: 3af43456e72d47398d2fbcf332489f55, State: Running, Role: LEADER
I20260806 16:05:40.456466 30229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af43456e72d47398d2fbcf332489f55" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46269 } }
I20260806 16:05:40.457103 30219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:05:40.457747 30229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af43456e72d47398d2fbcf332489f55. Latest consensus state: current_term: 1 leader_uuid: "3af43456e72d47398d2fbcf332489f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af43456e72d47398d2fbcf332489f55" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46269 } } }
I20260806 16:05:40.457867 30229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [sys.catalog]: This master's current role is: LEADER
I20260806 16:05:40.458250 30235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:05:40.458515 30231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af43456e72d47398d2fbcf332489f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af43456e72d47398d2fbcf332489f55" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46269 } } }
I20260806 16:05:40.458726 30231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55 [sys.catalog]: This master's current role is: LEADER
I20260806 16:05:40.462494 30235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:05:40.463138 30219 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 16:05:40.468772 30235 catalog_manager.cc:1347] Generated new cluster ID: c7eed0f44d404ef59cbc366542862168
I20260806 16:05:40.468904 30235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:05:40.523037 30235 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:05:40.524506 30235 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:05:40.540735 30235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55: Generated new TSK 0
I20260806 16:05:40.541838 30235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:05:41.293880 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46269
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:05:41.450567 30283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:05:41.450944 30283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:05:41.451014 30283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:05:41.457371 30283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:05:41.457460 30283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:05:41.457554 30283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:05:41.463783 30283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.148.20:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:05:41.464926 30283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:05:41.465991 30283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:05:41.478344 30295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:05:41.478456 30292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:05:41.478775 30291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:05:41.479233 30283 server_base.cc:1034] running on GCE node
I20260806 16:05:41.479681 30283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:05:41.480422 30283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:05:41.481602 30283 hybrid_clock.cc:648] HybridClock initialized: now 1786032341481561 us; error 52 us; skew 500 ppm
Aug 06 16:05:41 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:05:41.485387 30283 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:05:41.486657 30283 webserver.cc:460] Webserver started at http://127.13.148.1:36329/ using document root <none> and password file <none>
I20260806 16:05:41.487481 30283 fs_manager.cc:359] Metadata directory not provided
I20260806 16:05:41.487588 30283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:05:41.487895 30283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:05:41 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:05:41.505160 30283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "69596ab33c0a4049880129a726cfd9e6"
format_stamp: "Formatted at 2026-08-06 16:05:41 on dist-test-slave-7kzw"
server_key: "b78d1490780cd81b4f80b464cebc795f"
server_key_iv: "47a8610cb3d6defc68e7f5e693917ae6"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:41.505973 30283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "69596ab33c0a4049880129a726cfd9e6"
format_stamp: "Formatted at 2026-08-06 16:05:41 on dist-test-slave-7kzw"
server_key: "b78d1490780cd81b4f80b464cebc795f"
server_key_iv: "47a8610cb3d6defc68e7f5e693917ae6"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:41.511773 30283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 16:05:41.529871 30304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:41.531611 30283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:05:41.531770 30283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "69596ab33c0a4049880129a726cfd9e6"
format_stamp: "Formatted at 2026-08-06 16:05:41 on dist-test-slave-7kzw"
server_key: "b78d1490780cd81b4f80b464cebc795f"
server_key_iv: "47a8610cb3d6defc68e7f5e693917ae6"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:41.531932 30283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:05:41.560505 30283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:05:41.564581 30283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:05:41.564812 30283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:05:41.565589 30283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:05:41.566885 30283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:05:41.566962 30283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:41.567031 30283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:05:41.567073 30283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 16:05:41.578233 30283 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:45589
I20260806 16:05:41.579496 30283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:05:41.580118 30481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:45589 every 8 connection(s)
I20260806 16:05:41.589015 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 30283
I20260806 16:05:41.589164 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:05:41.589500 13904 external_mini_cluster.cc:1428] Setting key 9da73eba5226f23165aa9e4ee4965375
Aug 06 16:05:41 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:05:41.607824 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:41.590244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48543 (local address 127.13.148.62:46269)
0806 16:05:41.592368 (+ 2124us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:41.592372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:41.592387 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:05:41.592426 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:41.592429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:41.592519 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:41.592618 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:05:41.594308 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:41.595198 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:41.596973 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:41.597325 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:41.597409 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:41.602415 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:05:41.602420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:05:41.602422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:05:41.602454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:05:41.604398 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:41.605637 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:41.605642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:41.605644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:41.605758 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:41.606270 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:41.606273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:41.606275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:41.606563 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:05:41.606648 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:41.607479 (+ 831us) server_negotiation.cc:299] Negotiation successful
0806 16:05:41.607627 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2043,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:05:41.609826 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:41.588964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46269 (local address 127.13.148.1:48543)
0806 16:05:41.590820 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:41.590886 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:41.591943 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:41.592947 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:41.592956 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:41.593328 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:05:41.594122 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:41.594145 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:41.595360 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:41.595364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:41.596821 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:41.596833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:41.597658 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:41.597662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:41.597888 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:41.600196 (+ 2308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:05:41.600219 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:05:41.602156 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:05:41.604576 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:05:41.604586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:05:41.604609 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:05:41.605014 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:05:41.605898 (+ 884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:05:41.605901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:05:41.605903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:05:41.606122 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:05:41.606743 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:05:41.606751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:05:41.607119 (+ 368us) client_negotiation.cc:769] Sending connection context
0806 16:05:41.608291 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0806 16:05:41.608675 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"thread_start_us":191,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:05:41.611529 30482 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46269
I20260806 16:05:41.611801 30482 heartbeater.cc:464] Registering TS with master...
I20260806 16:05:41.612391 30482 heartbeater.cc:511] Master 127.13.148.62:46269 requested a full tablet report, sending...
I20260806 16:05:41.614334 30170 ts_manager.cc:194] Registered new tserver with Master: 69596ab33c0a4049880129a726cfd9e6 (127.13.148.1:45589)
I20260806 16:05:41.615942 30170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:48543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:05:41.634315 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46269
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:05:41 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 16:05:41.804850 30491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:05:41.805294 30491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:05:41.805374 30491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:05:41.811646 30491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:05:41.811748 30491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:05:41.811851 30491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:05:41.820060 30491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.148.20:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:05:41.821529 30491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:05:41.822986 30491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:05:41.836613 30506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:05:41.836794 30491 server_base.cc:1034] running on GCE node
W20260806 16:05:41.837163 30502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:05:41.837564 30504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:05:41.838084 30491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:05:41.838891 30491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:05:41.840106 30491 hybrid_clock.cc:648] HybridClock initialized: now 1786032341840079 us; error 42 us; skew 500 ppm
Aug 06 16:05:41 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:05:41.843863 30491 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:05:41.845085 30491 webserver.cc:460] Webserver started at http://127.13.148.2:44931/ using document root <none> and password file <none>
I20260806 16:05:41.845820 30491 fs_manager.cc:359] Metadata directory not provided
I20260806 16:05:41.845916 30491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:05:41.846171 30491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:05:41 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:05:41.862519 30491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "7c749efed4e04cc0b89d9f48d4e6ca18"
format_stamp: "Formatted at 2026-08-06 16:05:41 on dist-test-slave-7kzw"
server_key: "8d6c21077a6f07f3783177c8a1e0951c"
server_key_iv: "bb4dd1968702b57dada83e1216cbec75"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:41.863392 30491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "7c749efed4e04cc0b89d9f48d4e6ca18"
format_stamp: "Formatted at 2026-08-06 16:05:41 on dist-test-slave-7kzw"
server_key: "8d6c21077a6f07f3783177c8a1e0951c"
server_key_iv: "bb4dd1968702b57dada83e1216cbec75"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:41.870450 30491 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 16:05:41.888289 30514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:41.889909 30491 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260806 16:05:41.890069 30491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "7c749efed4e04cc0b89d9f48d4e6ca18"
format_stamp: "Formatted at 2026-08-06 16:05:41 on dist-test-slave-7kzw"
server_key: "8d6c21077a6f07f3783177c8a1e0951c"
server_key_iv: "bb4dd1968702b57dada83e1216cbec75"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:41.890296 30491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:05:41.924116 30491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:05:41.928650 30491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:05:41.928879 30491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:05:41.929682 30491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:05:41.930858 30491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:05:41.930939 30491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:41.931012 30491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:05:41.931047 30491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:41.943069 30491 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:37637
I20260806 16:05:41.943089 30669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:37637 every 8 connection(s)
I20260806 16:05:41.944046 30491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:05:41.947137 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 30491
I20260806 16:05:41.947288 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:05:41.947610 13904 external_mini_cluster.cc:1428] Setting key a7460b2d50452dd9521b5de28bcabf36
Aug 06 16:05:41 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:05:41.959645 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:41.946296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33283 (local address 127.13.148.62:46269)
0806 16:05:41.946489 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:41.946494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:41.947438 (+ 944us) server_negotiation.cc:407] Connection header received
0806 16:05:41.948554 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:41.948559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:41.948606 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:41.948706 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:05:41.949875 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:41.950402 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:41.951572 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:41.952005 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:41.952159 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:41.955223 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:05:41.955227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:05:41.955229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:05:41.955259 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:05:41.957116 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:41.957703 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:41.957706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:41.957707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:41.957790 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:41.958214 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:41.958219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:41.958221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:41.958562 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:05:41.958689 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:41.959261 (+ 572us) server_negotiation.cc:299] Negotiation successful
0806 16:05:41.959398 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:05:41.960450 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:41.946742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46269 (local address 127.13.148.2:33283)
0806 16:05:41.947282 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:41.947338 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:41.948353 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:41.948834 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:41.948840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:41.949214 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:05:41.949722 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:41.949736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:41.950555 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:41.950558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:41.951433 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:41.951442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:41.952189 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:41.952194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:41.952406 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:41.953089 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:05:41.953112 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:05:41.955023 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:05:41.957255 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:05:41.957260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:05:41.957273 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:05:41.957540 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:05:41.957936 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:05:41.957942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:05:41.957944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:05:41.958103 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:05:41.958689 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:05:41.958697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:05:41.958958 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 16:05:41.959213 (+ 255us) client_negotiation.cc:240] Negotiation successful
0806 16:05:41.959601 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":130,"threads_started":1}
I20260806 16:05:41.961826 30670 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46269
I20260806 16:05:41.962085 30670 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:05:41.962728 30670 heartbeater.cc:511] Master 127.13.148.62:46269 requested a full tablet report, sending...
I20260806 16:05:41.963855 30170 ts_manager.cc:194] Registered new tserver with Master: 7c749efed4e04cc0b89d9f48d4e6ca18 (127.13.148.2:37637)
I20260806 16:05:41.964505 30170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:33283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:05:41.985620 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:58399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46269
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:05:42 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
W20260806 16:05:42.170009 30675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:05:42.170403 30675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:05:42.170492 30675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:05:42.176858 30675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:05:42.176947 30675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:05:42.177040 30675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:05:42.185312 30675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.148.20:58399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:05:42.186940 30675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:05:42.188433 30675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:05:42.201848 30686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:05:42.201789 30688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:05:42.201808 30684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:05:42.202920 30675 server_base.cc:1034] running on GCE node
I20260806 16:05:42.203464 30675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:05:42.204178 30675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:05:42.205403 30675 hybrid_clock.cc:648] HybridClock initialized: now 1786032342205366 us; error 53 us; skew 500 ppm
Aug 06 16:05:42 dist-test-slave-7kzw krb5kdc[29502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032342, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:05:42.214709 30675 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:05:42.215953 30675 webserver.cc:460] Webserver started at http://127.13.148.3:45097/ using document root <none> and password file <none>
I20260806 16:05:42.216634 30675 fs_manager.cc:359] Metadata directory not provided
I20260806 16:05:42.216744 30675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:05:42.216998 30675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:05:42 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032342, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:05:42.233919 30675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "1d2f961f0a4248bbac49d448bf8fb86d"
format_stamp: "Formatted at 2026-08-06 16:05:42 on dist-test-slave-7kzw"
server_key: "6d2744748e76e9081eaed2eda35ce4a2"
server_key_iv: "21fd5a65e49a8a9fbc5ce05ea2c91e58"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:42.234758 30675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "1d2f961f0a4248bbac49d448bf8fb86d"
format_stamp: "Formatted at 2026-08-06 16:05:42 on dist-test-slave-7kzw"
server_key: "6d2744748e76e9081eaed2eda35ce4a2"
server_key_iv: "21fd5a65e49a8a9fbc5ce05ea2c91e58"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:42.240757 30675 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 16:05:42.259413 30698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:42.261469 30675 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 16:05:42.261619 30675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "1d2f961f0a4248bbac49d448bf8fb86d"
format_stamp: "Formatted at 2026-08-06 16:05:42 on dist-test-slave-7kzw"
server_key: "6d2744748e76e9081eaed2eda35ce4a2"
server_key_iv: "21fd5a65e49a8a9fbc5ce05ea2c91e58"
server_key_version: "kuduclusterkey@0"
I20260806 16:05:42.261770 30675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:05:42.279871 30675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:05:42.283798 30675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:05:42.283999 30675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:05:42.284693 30675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:05:42.285794 30675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:05:42.285861 30675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:42.285918 30675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:05:42.285996 30675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:05:42.299332 30675 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:37693
I20260806 16:05:42.299430 30849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:37693 every 8 connection(s)
I20260806 16:05:42.300631 30675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:05:42.310938 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 30675
I20260806 16:05:42.311090 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:05:42.311470 13904 external_mini_cluster.cc:1428] Setting key 470d6e5ea45cc3223484f8c78976ce88
Aug 06 16:05:42 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032342, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:05:42.318787 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.304719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:42311 (local address 127.13.148.62:46269)
0806 16:05:42.304854 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.304859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.304879 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:05:42.305487 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.305492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.305556 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.305670 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:05:42.307225 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.308159 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.309911 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.310314 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.310450 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.313870 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:05:42.313875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:05:42.313878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:05:42.313908 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:05:42.315956 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:42.316744 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:42.316749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:42.316751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:42.316854 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:42.317340 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:42.317343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:42.317345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:42.317591 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:05:42.317708 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.318434 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.318611 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:05:42.319926 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.303449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46269 (local address 127.13.148.3:42311)
0806 16:05:42.304071 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:42.304139 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:42.305224 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:42.305874 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:42.305884 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:42.306303 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:05:42.307039 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.307060 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.308316 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.308320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.309744 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.309757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.310523 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.310528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.310765 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.311507 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:05:42.311530 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:05:42.313664 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:05:42.316090 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:05:42.316097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:05:42.316125 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:05:42.316615 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:05:42.316981 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:05:42.316988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:05:42.316993 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:05:42.317212 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:05:42.317734 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:05:42.317742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:05:42.318096 (+ 354us) client_negotiation.cc:769] Sending connection context
0806 16:05:42.318356 (+ 260us) client_negotiation.cc:240] Negotiation successful
0806 16:05:42.318772 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":178,"threads_started":1}
I20260806 16:05:42.321295 30851 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46269
I20260806 16:05:42.321578 30851 heartbeater.cc:464] Registering TS with master...
I20260806 16:05:42.322201 30851 heartbeater.cc:511] Master 127.13.148.62:46269 requested a full tablet report, sending...
I20260806 16:05:42.323462 30170 ts_manager.cc:194] Registered new tserver with Master: 1d2f961f0a4248bbac49d448bf8fb86d (127.13.148.3:37693)
I20260806 16:05:42.324241 30170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:42311
I20260806 16:05:42.327505 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:05:42.340984 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.330151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.13.148.62:46269)
0806 16:05:42.330311 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.330316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.330733 (+ 417us) server_negotiation.cc:407] Connection header received
0806 16:05:42.330862 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.330866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.330915 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.330989 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:05:42.331986 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.332814 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.334216 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.334595 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.334706 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.336787 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:05:42.336792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:05:42.336795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:05:42.336823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:05:42.338934 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:42.339487 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:42.339491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:42.339493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:42.339587 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:05:42.339983 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:05:42.339986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:05:42.339988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:05:42.340225 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:05:42.340368 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.340677 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.340811 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:05:42.344959 30170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:05:42.350055 30170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:05:42.369994 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.361600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37637 (local address 127.0.0.1:59952)
0806 16:05:42.363302 (+ 1702us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:42.363359 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:42.363569 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:42.364322 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:42.364327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:42.364350 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:05:42.364499 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.364505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.365958 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.365961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.367725 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.367742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.369384 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.369387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.369537 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.369591 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 16:05:42.369702 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 16:05:42.369826 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1502,"mutex_wait_us":53,"thread_start_us":151,"threads_started":1}
I20260806 16:05:42.370687 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.362244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.13.148.2:37637)
0806 16:05:42.363003 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.363015 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.363455 (+ 440us) server_negotiation.cc:407] Connection header received
0806 16:05:42.363750 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.363757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.363972 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.364101 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:05:42.364626 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.365807 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.367957 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.369038 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.369121 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.369282 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.370384 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.370537 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":119,"threads_started":1}
I20260806 16:05:42.374483 30573 tablet_service.cc:1467] Processing CreateTablet for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900 (DEFAULT_TABLE table=test-table [id=b4d3fd1341bc4a629e706ac3cf3240fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:05:42.376426 30573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:05:42.376441 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.363619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45589 (local address 127.0.0.1:45472)
0806 16:05:42.366342 (+ 2723us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:42.366357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:42.366454 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:42.368364 (+ 1910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:42.368368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:42.368388 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:05:42.368492 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.368502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.371165 (+ 2663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.371168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.372477 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.372489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.375949 (+ 3460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.375954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.376066 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.376096 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 16:05:42.376196 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 16:05:42.376281 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2654,"thread_start_us":94,"threads_started":1}
I20260806 16:05:42.377024 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.364149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.13.148.1:45589)
0806 16:05:42.364792 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.364800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.366586 (+ 1786us) server_negotiation.cc:407] Connection header received
0806 16:05:42.367694 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.367702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.367963 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.368142 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:05:42.368651 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.370028 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.372644 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.373561 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.373686 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.373852 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.376740 (+ 2888us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.376862 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":126,"threads_started":1}
I20260806 16:05:42.380388 30364 tablet_service.cc:1467] Processing CreateTablet for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900 (DEFAULT_TABLE table=test-table [id=b4d3fd1341bc4a629e706ac3cf3240fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:05:42.381599 30364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:05:42.386636 30873 tablet_bootstrap.cc:492] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Bootstrap starting.
I20260806 16:05:42.389848 30873 tablet_bootstrap.cc:654] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Neither blocks nor log segments found. Creating new log.
I20260806 16:05:42.391347 30873 log.cc:824] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Log is configured to *not* fsync() on all Append() calls
I20260806 16:05:42.393244 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.363228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:37693 (local address 127.0.0.1:36928)
0806 16:05:42.364953 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:42.364970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:42.365070 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:42.365890 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:42.365894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:42.365915 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:05:42.366085 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.366093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.388581 (+ 22488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.388586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.390072 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.390087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.392755 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.392759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.392874 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.392907 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 16:05:42.392970 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 16:05:42.393061 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"mutex_wait_us":4,"thread_start_us":110,"threads_started":1}
I20260806 16:05:42.393653 30873 tablet_bootstrap.cc:492] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: No bootstrap required, opened a new log
I20260806 16:05:42.393858 30873 ts_tablet_manager.cc:1397] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:05:42.395231 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.362811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.13.148.3:37693)
0806 16:05:42.365366 (+ 2555us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.365381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.365405 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:05:42.365484 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.365489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.365674 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.373960 (+ 8286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:05:42.374500 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.375796 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.390375 (+ 14579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.391383 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.393995 (+ 2612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.394190 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.394253 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.394358 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":172,"threads_started":1}
I20260806 16:05:42.397617 30876 tablet_bootstrap.cc:492] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6: Bootstrap starting.
I20260806 16:05:42.398022 30873 raft_consensus.cc:348] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.398307 30873 raft_consensus.cc:374] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:05:42.398361 30873 raft_consensus.cc:729] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c749efed4e04cc0b89d9f48d4e6ca18, State: Initialized, Role: FOLLOWER
I20260806 16:05:42.400094 30873 consensus_queue.cc:260] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.401152 30873 ts_tablet_manager.cc:1428] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 16:05:42.401530 30670 heartbeater.cc:503] Master 127.13.148.62:46269 was elected leader, sending a full tablet report...
I20260806 16:05:42.401952 30743 tablet_service.cc:1467] Processing CreateTablet for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900 (DEFAULT_TABLE table=test-table [id=b4d3fd1341bc4a629e706ac3cf3240fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:05:42.403381 30743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:05:42.406512 30876 tablet_bootstrap.cc:654] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6: Neither blocks nor log segments found. Creating new log.
I20260806 16:05:42.408236 30876 log.cc:824] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6: Log is configured to *not* fsync() on all Append() calls
I20260806 16:05:42.410929 30876 tablet_bootstrap.cc:492] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6: No bootstrap required, opened a new log
I20260806 16:05:42.411144 30876 ts_tablet_manager.cc:1397] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260806 16:05:42.412930 30881 tablet_bootstrap.cc:492] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Bootstrap starting.
I20260806 16:05:42.416062 30881 tablet_bootstrap.cc:654] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Neither blocks nor log segments found. Creating new log.
I20260806 16:05:42.417081 30881 log.cc:824] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Log is configured to *not* fsync() on all Append() calls
I20260806 16:05:42.421278 30876 raft_consensus.cc:348] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.421563 30876 raft_consensus.cc:374] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:05:42.421679 30876 raft_consensus.cc:729] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69596ab33c0a4049880129a726cfd9e6, State: Initialized, Role: FOLLOWER
I20260806 16:05:42.422219 30876 consensus_queue.cc:260] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.423977 30876 ts_tablet_manager.cc:1428] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20260806 16:05:42.424435 30482 heartbeater.cc:503] Master 127.13.148.62:46269 was elected leader, sending a full tablet report...
I20260806 16:05:42.443843 30881 tablet_bootstrap.cc:492] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: No bootstrap required, opened a new log
I20260806 16:05:42.444227 30881 ts_tablet_manager.cc:1397] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.005s
W20260806 16:05:42.445436 30671 tablet.cc:2369] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:05:42.449064 30881 raft_consensus.cc:348] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.449540 30881 raft_consensus.cc:374] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:05:42.449668 30881 raft_consensus.cc:729] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2f961f0a4248bbac49d448bf8fb86d, State: Initialized, Role: FOLLOWER
I20260806 16:05:42.450515 30881 consensus_queue.cc:260] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.451826 30881 ts_tablet_manager.cc:1428] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260806 16:05:42.452270 30851 heartbeater.cc:503] Master 127.13.148.62:46269 was elected leader, sending a full tablet report...
Aug 06 16:05:42 dist-test-slave-7kzw krb5kdc[29502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032341, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
W20260806 16:05:42.552960 30852 tablet.cc:2369] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:05:42.581207 30483 tablet.cc:2369] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:05:42.720599 30886 raft_consensus.cc:482] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:05:42.721024 30886 raft_consensus.cc:504] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.723573 30886 leader_election.cc:288] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c749efed4e04cc0b89d9f48d4e6ca18 (127.13.148.2:37637), 69596ab33c0a4049880129a726cfd9e6 (127.13.148.1:45589)
I20260806 16:05:42.734658 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.726125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:60993 (local address 127.13.148.2:37637)
0806 16:05:42.726628 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.726642 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.726660 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:05:42.726754 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.726758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.726851 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.726992 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:05:42.728037 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.729243 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.731808 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.732785 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.733038 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.733174 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.734302 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.734429 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260806 16:05:42.734017 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.724283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37637 (local address 127.13.148.3:60993)
0806 16:05:42.725484 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:42.725517 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:42.725658 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:42.727497 (+ 1839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:42.727504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:42.727536 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:05:42.727739 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.727758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.729615 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.729619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.731533 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.731564 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.733288 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.733293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.733484 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.733532 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 16:05:42.733645 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 16:05:42.733779 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074}
I20260806 16:05:42.736958 30603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1acdbb92d24f3aa6cbab3bdf7d2900" candidate_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" is_pre_election: true
I20260806 16:05:42.737520 30603 raft_consensus.cc:2454] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2f961f0a4248bbac49d448bf8fb86d in term 0.
I20260806 16:05:42.738385 30883 raft_consensus.cc:482] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:05:42.738767 30883 raft_consensus.cc:504] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.739080 30701 leader_election.cc:302] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2f961f0a4248bbac49d448bf8fb86d, 7c749efed4e04cc0b89d9f48d4e6ca18; no voters:
I20260806 16:05:42.739782 30886 raft_consensus.cc:2781] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:05:42.740039 30886 raft_consensus.cc:482] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:05:42.740221 30886 raft_consensus.cc:3037] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:05:42.741258 30883 leader_election.cc:288] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c749efed4e04cc0b89d9f48d4e6ca18 (127.13.148.2:37637), 1d2f961f0a4248bbac49d448bf8fb86d (127.13.148.3:37693)
I20260806 16:05:42.744688 30886 raft_consensus.cc:504] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.749547 30603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1acdbb92d24f3aa6cbab3bdf7d2900" candidate_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18"
I20260806 16:05:42.749776 30603 raft_consensus.cc:3037] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:05:42.752003 30603 raft_consensus.cc:2454] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2f961f0a4248bbac49d448bf8fb86d in term 1.
I20260806 16:05:42.752754 30701 leader_election.cc:302] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d2f961f0a4248bbac49d448bf8fb86d, 7c749efed4e04cc0b89d9f48d4e6ca18; no voters:
I20260806 16:05:42.757074 30886 leader_election.cc:288] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [CANDIDATE]: Term 1 election: Requested vote from peers 7c749efed4e04cc0b89d9f48d4e6ca18 (127.13.148.2:37637), 69596ab33c0a4049880129a726cfd9e6 (127.13.148.1:45589)
I20260806 16:05:42.757653 30886 raft_consensus.cc:2781] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:05:42.758394 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.726357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:42181 (local address 127.13.148.1:45589)
0806 16:05:42.727325 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.727340 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.735931 (+ 8591us) server_negotiation.cc:407] Connection header received
0806 16:05:42.742486 (+ 6555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.742493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.742577 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.742652 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:05:42.745069 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.746321 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.753876 (+ 7555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.754863 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.755112 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.755226 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.757959 (+ 2733us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.758106 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":777}
I20260806 16:05:42.759184 30886 raft_consensus.cc:686] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 1 LEADER]: Becoming Leader. State: Replica: 1d2f961f0a4248bbac49d448bf8fb86d, State: Running, Role: LEADER
I20260806 16:05:42.759909 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.724762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45589 (local address 127.13.148.3:42181)
0806 16:05:42.735797 (+ 11035us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:42.735823 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:42.741699 (+ 5876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:42.742877 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:42.742881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:42.742901 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:05:42.743058 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.743078 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.746609 (+ 3531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.746612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.748243 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.748256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.755443 (+ 7187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.755447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.755607 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.755662 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 16:05:42.759415 (+ 3753us) client_negotiation.cc:240] Negotiation successful
0806 16:05:42.759650 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10932,"thread_start_us":192,"threads_started":1}
I20260806 16:05:42.764554 30379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1acdbb92d24f3aa6cbab3bdf7d2900" candidate_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69596ab33c0a4049880129a726cfd9e6" is_pre_election: true
I20260806 16:05:42.764971 30379 raft_consensus.cc:2454] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d2f961f0a4248bbac49d448bf8fb86d in term 0.
I20260806 16:05:42.765520 30382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1acdbb92d24f3aa6cbab3bdf7d2900" candidate_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69596ab33c0a4049880129a726cfd9e6"
I20260806 16:05:42.765700 30382 raft_consensus.cc:3037] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:05:42.768203 30382 raft_consensus.cc:2454] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d2f961f0a4248bbac49d448bf8fb86d in term 1.
I20260806 16:05:42.769855 30886 consensus_queue.cc:237] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } }
I20260806 16:05:42.772384 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.742886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:56825 (local address 127.13.148.3:37693)
0806 16:05:42.743129 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.743136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.751750 (+ 8614us) server_negotiation.cc:407] Connection header received
0806 16:05:42.752127 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.752145 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.752242 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.752368 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:05:42.753247 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.754435 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.768307 (+ 13872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.769485 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.769721 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.769828 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.771988 (+ 2160us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.772150 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 16:05:42.773980 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.742740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:37693 (local address 127.13.148.1:56825)
0806 16:05:42.751555 (+ 8815us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:42.751607 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:42.751866 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:42.752730 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:42.752734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:42.752765 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:05:42.752972 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.752990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.755582 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.755600 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.762436 (+ 6836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.762491 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.771213 (+ 8722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.771219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.771428 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.771504 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 16:05:42.773041 (+ 1537us) client_negotiation.cc:240] Negotiation successful
0806 16:05:42.773222 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8684,"thread_start_us":183,"threads_started":1}
I20260806 16:05:42.775484 30763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1acdbb92d24f3aa6cbab3bdf7d2900" candidate_uuid: "69596ab33c0a4049880129a726cfd9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" is_pre_election: true
I20260806 16:05:42.778071 30170 catalog_manager.cc:5495] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d2f961f0a4248bbac49d448bf8fb86d (127.13.148.3). New cstate: current_term: 1 leader_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d2f961f0a4248bbac49d448bf8fb86d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 } health_report { overall_health: UNKNOWN } } }
I20260806 16:05:42.784773 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.742090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:44115 (local address 127.13.148.2:37637)
0806 16:05:42.742309 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.742313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.753372 (+ 11059us) server_negotiation.cc:407] Connection header received
0806 16:05:42.753455 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.753467 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.753537 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.753674 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:05:42.755834 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.756864 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.761622 (+ 4758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.762692 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.762907 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.762992 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.784356 (+ 21364us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.784548 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 16:05:42.785863 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.741944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37637 (local address 127.13.148.1:44115)
0806 16:05:42.750845 (+ 8901us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:42.750889 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:05:42.751105 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:05:42.754704 (+ 3599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:05:42.754709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:05:42.754753 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:05:42.755285 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.755294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.757123 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.757127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.761012 (+ 3885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:05:42.761040 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.783752 (+ 22712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.783759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:05:42.784013 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.784068 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 16:05:42.785371 (+ 1303us) client_negotiation.cc:240] Negotiation successful
0806 16:05:42.785575 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8734,"mutex_wait_us":1,"thread_start_us":263,"threads_started":1}
I20260806 16:05:42.786870 30603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1acdbb92d24f3aa6cbab3bdf7d2900" candidate_uuid: "69596ab33c0a4049880129a726cfd9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" is_pre_election: true
I20260806 16:05:42.787242 30603 raft_consensus.cc:2379] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69596ab33c0a4049880129a726cfd9e6 in current term 1: Already voted for candidate 1d2f961f0a4248bbac49d448bf8fb86d in this term.
I20260806 16:05:42.788257 30309 leader_election.cc:302] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69596ab33c0a4049880129a726cfd9e6; no voters: 1d2f961f0a4248bbac49d448bf8fb86d, 7c749efed4e04cc0b89d9f48d4e6ca18
I20260806 16:05:42.789225 30883 raft_consensus.cc:2726] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 16:05:42.842270 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:42.835610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.13.148.3:37693)
0806 16:05:42.836347 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0806 16:05:42.836352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:05:42.836376 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:05:42.836508 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:05:42.836521 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:05:42.836645 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:05:42.836846 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:05:42.837266 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.838223 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.839685 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:05:42.840006 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:05:42.840214 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:05:42.840724 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:05:42.841561 (+ 837us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:05:42.841798 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 16:05:42.841915 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 16:05:42.842015 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599}
I20260806 16:05:42.856901 30603 raft_consensus.cc:1261] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 1 FOLLOWER]: Refusing update from remote peer 1d2f961f0a4248bbac49d448bf8fb86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:05:42.857038 30382 raft_consensus.cc:1261] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 69596ab33c0a4049880129a726cfd9e6 [term 1 FOLLOWER]: Refusing update from remote peer 1d2f961f0a4248bbac49d448bf8fb86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:05:42.858382 30894 consensus_queue.cc:1035] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69596ab33c0a4049880129a726cfd9e6" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:05:42.862104 30886 consensus_queue.cc:1035] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c749efed4e04cc0b89d9f48d4e6ca18" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:05:42.877482 30907 mvcc.cc:205] Tried to move back new op lower bound from 7315588476327022592 to 7315588475992637440. Current Snapshot: MvccSnapshot[applied={T|T < 7315588476327022592}]
I20260806 16:05:42.896637 30913 mvcc.cc:205] Tried to move back new op lower bound from 7315588476327022592 to 7315588475992637440. Current Snapshot: MvccSnapshot[applied={T|T < 7315588476327022592}]
I20260806 16:05:42.907011 30168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:05:42.907474 30168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:05:42.912997 30168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3af43456e72d47398d2fbcf332489f55: Sending DeleteTablet for 3 replicas of tablet bc1acdbb92d24f3aa6cbab3bdf7d2900
I20260806 16:05:42.914318 30573 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:05:42 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:59952
I20260806 16:05:42.915980 30743 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:05:42 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36928
I20260806 16:05:42.917454 30364 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:05:42 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:45472
I20260806 16:05:42.918694 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 30283
I20260806 16:05:42.918772 30919 tablet_replica.cc:330] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: stopping tablet replica
I20260806 16:05:42.919310 30919 raft_consensus.cc:2229] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:05:42.929258 30919 raft_consensus.cc:2258] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:05:42.929738 30920 tablet_replica.cc:330] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: stopping tablet replica
I20260806 16:05:42.932869 30919 ts_tablet_manager.cc:1905] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:05:42.933173 30920 raft_consensus.cc:2229] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:05:42.933980 30920 raft_consensus.cc:2258] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:05:42.937474 30920 ts_tablet_manager.cc:1905] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:05:42.943235 30919 ts_tablet_manager.cc:1918] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:05:42.956569 30920 ts_tablet_manager.cc:1918] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:05:42.956961 30919 log.cc:1196] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/bc1acdbb92d24f3aa6cbab3bdf7d2900
I20260806 16:05:42.957861 30919 ts_tablet_manager.cc:1939] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 7c749efed4e04cc0b89d9f48d4e6ca18: Deleting consensus metadata
I20260806 16:05:42.958600 30920 log.cc:1196] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/bc1acdbb92d24f3aa6cbab3bdf7d2900
I20260806 16:05:42.959345 30920 ts_tablet_manager.cc:1939] T bc1acdbb92d24f3aa6cbab3bdf7d2900 P 1d2f961f0a4248bbac49d448bf8fb86d: Deleting consensus metadata
I20260806 16:05:42.961683 30147 catalog_manager.cc:4841] TS 7c749efed4e04cc0b89d9f48d4e6ca18 (127.13.148.2:37637): tablet bc1acdbb92d24f3aa6cbab3bdf7d2900 (table test-table [id=b4d3fd1341bc4a629e706ac3cf3240fe]) successfully deleted
I20260806 16:05:42.964712 30147 catalog_manager.cc:4841] TS 1d2f961f0a4248bbac49d448bf8fb86d (127.13.148.3:37693): tablet bc1acdbb92d24f3aa6cbab3bdf7d2900 (table test-table [id=b4d3fd1341bc4a629e706ac3cf3240fe]) successfully deleted
W20260806 16:05:42.992226 30147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:45589 (error 108)
I20260806 16:05:42.994524 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 30491
I20260806 16:05:43.027266 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:05:43.015616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45589 (local address 127.0.0.1:45474)
0806 16:05:43.026816 (+ 11200us) negotiation.cc:107] Waiting for socket to connect
0806 16:05:43.027023 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10993,"thread_start_us":128,"threads_started":1}
W20260806 16:05:43.027817 30147 catalog_manager.cc:4584] TS 69596ab33c0a4049880129a726cfd9e6 (127.13.148.1:45589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc1acdbb92d24f3aa6cbab3bdf7d2900: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45589: connect: Connection refused (error 111)
I20260806 16:05:43.034198 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 30675
I20260806 16:05:43.056890 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 30121
I20260806 16:05:43.079592 13904 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 16:05:43.435534 13904 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 16:05:43.435730 13904 mini_ranger.cc:66] Stopping Ranger...
I20260806 16:05:43.780932 13904 mini_ranger.cc:68] Stopped Ranger
2026-08-06 16:05:43.781 UTC [29539] LOG: received smart shutdown request
2026-08-06 16:05:43.785 UTC [29539] LOG: background worker "logical replication launcher" (PID 29550) exited with exit code 1
2026-08-06 16:05:43.785 UTC [29545] LOG: shutting down
2026-08-06 16:05:43.966 UTC [29539] LOG: database system is shut down
2026-08-06T16:05:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49514 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:05:44 dist-test-slave-7kzw krb5kdc[30934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:05:44 dist-test-slave-7kzw krb5kdc[30934](info): set up 2 sockets
Aug 06 16:05:44 dist-test-slave-7kzw krb5kdc[30934](info): commencing operation
krb5kdc: starting...
W20260806 16:05:45.391655 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:05:45 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:05:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/rangeradmin_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/rangeradmin_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/rangerlookup_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/rangerlookup_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
I20260806 16:05:45.590072 13904 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwTlxSD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/postgres -l logfile start
2026-08-06 16:05:48.329 UTC [30972] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 16:05:48.329 UTC [30972] LOG: listening on IPv4 address "127.13.148.20", port 53099
2026-08-06 16:05:48.333 UTC [30972] LOG: listening on Unix socket "/tmp/.s.PGSQL.53099"
2026-08-06 16:05:48.377 UTC [30976] LOG: database system was shut down at 2026-08-06 16:05:46 UTC
2026-08-06 16:05:48.381 UTC [30972] LOG: database system is ready to accept connections
I20260806 16:05:49.523519 13904 mini_postgres.cc:96] Postgres bound to 53099
2026-08-06 16:05:49.531 UTC [30987] FATAL: database "slave" does not exist
127.13.148.20:53099 - accepting connections
I20260806 16:05:49.532122 13904 mini_ranger.cc:161] Starting Ranger...
I20260806 16:05:49.548202 13904 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 16:05:50.077430 13904 mini_ranger.cc:87] Created ranger Postgres database
I20260806 16:05:50.077540 13904 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-admin
2026-08-06 16:05:50,356 [I] DB FLAVOR :POSTGRES
2026-08-06 16:05:50,357 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 16:05:50,357 [I] Checking connection..
2026-08-06 16:05:50,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 16:05:50,651 [I] Checking connection passed.
2026-08-06 16:05:50,651 [I] --------- Verifying version history table ---------
2026-08-06 16:05:50,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:05:50,900 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 16:05:50,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:05:51,150 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 16:05:51,150 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 16:05:51,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 16:05:51.396 UTC [31083] WARNING: there is no transaction in progress
2026-08-06 16:05:51,405 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 16:05:51,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 16:05:51,705 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 16:05:51,705 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 16:05:51,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 16:05:51,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 16:05:52,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7kzw.c.gcp-upstream.internal','N') ;"
2026-08-06 16:05:52,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 16:05:52,746 [I] Table x_portal_user does not exist in database ranger
2026-08-06 16:05:52,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 16:05:52,993 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 16:05:52,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 16:05:53,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 16:05:53,554 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 16:05:53,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 16:05:54.577 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.594 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.621 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.644 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.663 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.682 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.744 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.751 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.768 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.779 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.790 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.805 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.816 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.823 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.831 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:54.840 UTC [31237] WARNING: there is no transaction in progress
2026-08-06 16:05:55,560 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 16:05:55,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 16:05:55,816 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 16:05:55,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 16:05:56,061 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 16:05:56,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 16:05:56,303 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 16:05:56,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 16:05:56,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 16:05:56,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7kzw.c.gcp-upstream.internal';"
2026-08-06 16:05:57,058 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 16:05:57,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 16:05:57,322 [I] DB_PATCHES have already been applied
I20260806 16:05:57.328688 13904 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwTlxSD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 16:05:57.328761 13904 mini_ranger.cc:193] Using host: 127.13.148.20
I20260806 16:05:57.333192 13904 mini_ranger.cc:227] Ranger admin URL: http://127.13.148.20:52943
Aug 06, 2026 4:05:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:05:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:05:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:05:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:05:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:05:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:05:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 16:05:58.650285 13904 mini_ranger.cc:160] Time spent starting Ranger: real 9.118s user 0.007s sys 0.006s
Aug 06, 2026 4:06:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:06:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 4:06:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:06:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:06:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 16:06:21.111537 13904 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/rangerkms_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/rangerkms_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.148.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 16:06:21 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032381, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 16:06:21.201272 13904 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 16:06:21.212091 13904 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 16:06:22.118808 13904 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 16:06:22.118924 13904 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms
I20260806 16:06:22.118947 13904 mini_ranger_kms.cc:227] Using postgres at 127.13.148.20:53099
2026-08-06 16:06:22,309 [I] DB FLAVOR :POSTGRES
2026-08-06 16:06:22,309 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 16:06:22,310 [I] Checking connection
2026-08-06 16:06:22,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 16:06:22,553 [I] connection success
2026-08-06 16:06:22,554 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 16:06:22,554 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 16:06:22,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 16:06:22,812 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 16:06:22,812 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 16:06:22,812 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 16:06:22,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:53099/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 16:06:23,085 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 16:06:23.578595 13904 mini_ranger_kms.cc:310] Created kms service
I20260806 16:06:23.786780 13904 mini_ranger_kms.cc:326] Created kudu user
I20260806 16:06:23.831722 13904 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 16:06:23 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032381, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:24.269905 13904 mini_ranger_kms.cc:384] Added ranger policy
I20260806 16:06:24.270061 13904 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwTlxSD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwTlxSD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 16:06:24.270083 13904 mini_ranger_kms.cc:240] Using host: 127.13.148.20
I20260806 16:06:24.274928 13904 mini_ranger_kms.cc:276] Ranger KMS PID: 31491
I20260806 16:06:24.275015 13904 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.148.20:42001
Aug 06, 2026 4:06:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:06:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:06:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:06:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:06:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:06:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 16:06:26 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032386, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:26.961400 13904 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.760s user 0.000s sys 0.173s
I20260806 16:06:26.961537 13904 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 16:06:26.961611 13904 mini_ranger_kms.cc:401] 127.13.148.20:42001/kms/v1/keys
Aug 06 16:06:27 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032386, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Aug 06 16:06:27 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032386, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Aug 06 16:06:29 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:06:29.867796 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36291
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36291
--ranger_config_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:42001/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:29.985630 31554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:29.985954 31554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:29.986035 31554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:29.990172 31554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:29.990272 31554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:29.990312 31554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:29.990372 31554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:29.990410 31554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:29.996107 31554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.148.20:42001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36291
--ranger_config_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:29.997222 31554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:29.998297 31554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:30.007668 31562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:30.007573 31565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:30.007715 31554 server_base.cc:1034] running on GCE node
W20260806 16:06:30.007596 31567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:30.008445 31554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:30.009667 31554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:30.010874 31554 hybrid_clock.cc:648] HybridClock initialized: now 1786032390010843 us; error 46 us; skew 500 ppm
Aug 06 16:06:30 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032390, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:30.018321 31554 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:06:30.019382 31554 webserver.cc:460] Webserver started at http://127.13.148.62:35755/ using document root <none> and password file <none>
I20260806 16:06:30.020038 31554 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:30.020131 31554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:30.020391 31554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:06:30 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032390, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:30.077006 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "ffbee2b3100243cd8441104a101a5d7f"
format_stamp: "Formatted at 2026-08-06 16:06:30 on dist-test-slave-7kzw"
server_key: "88fb95c07845d326ae0b9edc7b36ae29"
server_key_iv: "108692f94df4b7ca72013ad8badb1ed8"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:30.077680 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "ffbee2b3100243cd8441104a101a5d7f"
format_stamp: "Formatted at 2026-08-06 16:06:30 on dist-test-slave-7kzw"
server_key: "88fb95c07845d326ae0b9edc7b36ae29"
server_key_iv: "108692f94df4b7ca72013ad8badb1ed8"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:30.081887 31554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:06:30.096959 31576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:30.098543 31554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:06:30.098722 31554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "ffbee2b3100243cd8441104a101a5d7f"
format_stamp: "Formatted at 2026-08-06 16:06:30 on dist-test-slave-7kzw"
server_key: "88fb95c07845d326ae0b9edc7b36ae29"
server_key_iv: "108692f94df4b7ca72013ad8badb1ed8"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:30.098889 31554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:30.126078 31554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:30.129405 31554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:30.129633 31554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:30.138490 31554 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36291
I20260806 16:06:30.138500 31649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36291 every 8 connection(s)
I20260806 16:06:30.139492 31554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:06:30.142752 31651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:30.147707 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 31554
I20260806 16:06:30.147826 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:06:30.148113 13904 external_mini_cluster.cc:1428] Setting key a2d1bfea526ff90c8421b4f6511c8403
I20260806 16:06:30.149396 31651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Bootstrap starting.
I20260806 16:06:30.151875 31651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:30.152668 31651 log.cc:824] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:30.154861 31651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: No bootstrap required, opened a new log
Aug 06 16:06:30 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:30.159112 31651 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } }
I20260806 16:06:30.159368 31651 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:30.159452 31651 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbee2b3100243cd8441104a101a5d7f, State: Initialized, Role: FOLLOWER
I20260806 16:06:30.159899 31651 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } }
I20260806 16:06:30.160053 31651 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:06:30.160145 31651 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:06:30.160267 31651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:30.161289 31651 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } }
I20260806 16:06:30.161641 31651 leader_election.cc:302] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffbee2b3100243cd8441104a101a5d7f; no voters:
I20260806 16:06:30.162007 31651 leader_election.cc:288] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:06:30.162107 31658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:06:30.162323 31658 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 1 LEADER]: Becoming Leader. State: Replica: ffbee2b3100243cd8441104a101a5d7f, State: Running, Role: LEADER
I20260806 16:06:30.162703 31658 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } }
I20260806 16:06:30.163384 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:30.149708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.13.148.62:36291)
0806 16:06:30.150184 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:30.150203 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:30.150239 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:06:30.150985 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:30.151004 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:30.151345 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:30.151692 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:30.152511 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:30.153515 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:30.154388 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:30.154937 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:30.155114 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:30.158098 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:30.158103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:30.158116 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:30.158143 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:30.160108 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:30.160619 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:30.160625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:30.160631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:30.160753 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:30.161049 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:30.161053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:30.161055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:30.161499 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:30.161712 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:30.161994 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 16:06:30.162538 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":152,"threads_started":1}
I20260806 16:06:30.163398 31651 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:06:30.165798 31651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 16:06:30.166254 31660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffbee2b3100243cd8441104a101a5d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } } }
I20260806 16:06:30.166399 31660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:30.166851 31682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:06:30.167033 31662 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffbee2b3100243cd8441104a101a5d7f. Latest consensus state: current_term: 1 leader_uuid: "ffbee2b3100243cd8441104a101a5d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } } }
I20260806 16:06:30.167147 31662 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:30.172359 31682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:06:30.179461 31682 catalog_manager.cc:1347] Generated new cluster ID: c17fe94255924f6cb73232b10be27a69
I20260806 16:06:30.179536 31682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:06:30.195662 31682 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:06:30.196801 31682 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:06:30.202239 31682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Generated new TSK 0
I20260806 16:06:30.203227 31682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:06:30.907049 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:42001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36291
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:31.038748 31714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:31.039112 31714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:31.039180 31714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:31.045339 31714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:31.045428 31714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:31.045521 31714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:06:31.053625 31714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.148.20:42001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:31.055220 31714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:31.056524 31714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:31.068732 31723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:31.069227 31726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:31.069531 31724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:31.070216 31714 server_base.cc:1034] running on GCE node
I20260806 16:06:31.070724 31714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:31.071369 31714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:31.078768 31714 hybrid_clock.cc:648] HybridClock initialized: now 1786032391078727 us; error 52 us; skew 500 ppm
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:31.081696 31714 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:06:31.082767 31714 webserver.cc:460] Webserver started at http://127.13.148.1:42083/ using document root <none> and password file <none>
I20260806 16:06:31.083352 31714 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:31.083447 31714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:31.083678 31714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:31.106969 31714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "2dbb8570c146410caffc4b5a28a57cc7"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "d92779eb34e772f103de3d746987254b"
server_key_iv: "3261048d2f4a40d64cf21484d2f9e979"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.107802 31714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "2dbb8570c146410caffc4b5a28a57cc7"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "d92779eb34e772f103de3d746987254b"
server_key_iv: "3261048d2f4a40d64cf21484d2f9e979"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.113281 31714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 16:06:31.135828 31735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:31.137605 31714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 16:06:31.137748 31714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "2dbb8570c146410caffc4b5a28a57cc7"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "d92779eb34e772f103de3d746987254b"
server_key_iv: "3261048d2f4a40d64cf21484d2f9e979"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.137880 31714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:31.170677 31714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:31.174757 31714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:31.175035 31714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:31.175807 31714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:31.176967 31714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:31.177085 31714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:31.177238 31714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:31.177340 31714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:31.191864 31714 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:41241
I20260806 16:06:31.193189 31714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:06:31.193853 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 31714
I20260806 16:06:31.193982 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:06:31.194303 13904 external_mini_cluster.cc:1428] Setting key f30d53c11ecd58db29f4175e43ad0f61
I20260806 16:06:31.194941 31898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:41241 every 8 connection(s)
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:31.230708 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:31.198349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36291 (local address 127.13.148.1:51221)
0806 16:06:31.202174 (+ 3825us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:31.202238 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:31.207289 (+ 5051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:31.208498 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:31.208511 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:31.208925 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:31.209669 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:31.209701 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.210963 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.210967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:31.212390 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:31.212403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.215370 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.215374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:31.215581 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:31.222642 (+ 7061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:31.222667 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:31.224568 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:31.227074 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:31.227081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:31.227101 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:31.227503 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:31.227867 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:31.227870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:31.227872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:31.228068 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:31.228580 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:31.228587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:31.228946 (+ 359us) client_negotiation.cc:769] Sending connection context
0806 16:06:31.229190 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 16:06:31.229555 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3564,"thread_start_us":155,"threads_started":1}
I20260806 16:06:31.231462 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:31.198609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:51221 (local address 127.13.148.62:36291)
0806 16:06:31.207883 (+ 9274us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:31.207892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:31.207910 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:06:31.207958 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:31.207961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:31.208016 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:31.208123 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:31.209875 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.210801 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.212539 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.212920 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.213025 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:31.224838 (+ 11813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:31.224843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:31.224846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:31.224880 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:31.226916 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:31.227636 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:31.227639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:31.227641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:31.227754 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:31.228184 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:31.228188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:31.228189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:31.228427 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:31.228523 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:31.231160 (+ 2637us) server_negotiation.cc:299] Negotiation successful
0806 16:06:31.231296 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9161,"thread_start_us":120,"threads_started":1}
I20260806 16:06:31.233358 31899 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36291
I20260806 16:06:31.233639 31899 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:31.234150 31899 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:31.236045 31602 ts_manager.cc:194] Registered new tserver with Master: 2dbb8570c146410caffc4b5a28a57cc7 (127.13.148.1:41241)
I20260806 16:06:31.237632 31602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:51221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:06:31.250569 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:42001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36291
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 16:06:31.413497 31905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:31.414355 31905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:31.414507 31905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:31.420804 31905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:31.421000 31905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:31.421142 31905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:06:31.429211 31905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.148.20:42001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:31.431044 31905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:31.432510 31905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:31.445513 31916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:31.445523 31911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:31.445899 31912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:31.447203 31905 server_base.cc:1034] running on GCE node
I20260806 16:06:31.447818 31905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:31.448846 31905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:31.450101 31905 hybrid_clock.cc:648] HybridClock initialized: now 1786032391450071 us; error 43 us; skew 500 ppm
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:31.454156 31905 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:06:31.455264 31905 webserver.cc:460] Webserver started at http://127.13.148.2:36421/ using document root <none> and password file <none>
I20260806 16:06:31.455986 31905 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:31.456085 31905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:31.456404 31905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:31.472010 31905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "9913a4a033f24950af4daf47047e75ca"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "9080a12ddd37bfe2fa59ed94aee9c325"
server_key_iv: "1fee781a8d456751fbc5c1da5630181a"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.473420 31905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "9913a4a033f24950af4daf47047e75ca"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "9080a12ddd37bfe2fa59ed94aee9c325"
server_key_iv: "1fee781a8d456751fbc5c1da5630181a"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.478983 31905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 16:06:31.512713 31924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:31.514699 31905 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.000s
I20260806 16:06:31.514928 31905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "9913a4a033f24950af4daf47047e75ca"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "9080a12ddd37bfe2fa59ed94aee9c325"
server_key_iv: "1fee781a8d456751fbc5c1da5630181a"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.515188 31905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:31.554980 31905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:31.558918 31905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:31.559132 31905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:31.559813 31905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:31.560925 31905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:31.560990 31905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:31.561048 31905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:31.561169 31905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 16:06:31.573570 31905 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:37515
I20260806 16:06:31.574841 31905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:06:31.575776 32088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:37515 every 8 connection(s)
I20260806 16:06:31.583511 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 31905
I20260806 16:06:31.583642 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:06:31.583941 13904 external_mini_cluster.cc:1428] Setting key baaa8b07f71d95c8d073c7be84c3e90f
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:31.594564 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:31.579396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:37871 (local address 127.13.148.62:36291)
0806 16:06:31.579527 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:31.579532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:31.580332 (+ 800us) server_negotiation.cc:407] Connection header received
0806 16:06:31.581541 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:31.581545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:31.581603 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:31.581657 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:31.583275 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.584137 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.586701 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.587039 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.587139 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:31.590241 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:31.590245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:31.590248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:31.590275 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:31.592054 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:31.592661 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:31.592665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:31.592666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:31.592753 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:31.593171 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:31.593174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:31.593176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:31.593463 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:31.593552 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:31.594220 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 16:06:31.594352 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:06:31.596406 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:31.578291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36291 (local address 127.13.148.2:37871)
0806 16:06:31.580175 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:31.580229 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:31.581292 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:31.582049 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:31.582069 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:31.582420 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:31.583088 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:31.583106 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.584307 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.584310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:31.586559 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:31.586571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.587372 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.587376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:31.587568 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:31.588267 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:31.588289 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:31.590037 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:31.592204 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:31.592210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:31.592226 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:31.592544 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:31.592858 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:31.592861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:31.592863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:31.593045 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:31.593626 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:31.593633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:31.593934 (+ 301us) client_negotiation.cc:769] Sending connection context
0806 16:06:31.595014 (+ 1080us) client_negotiation.cc:240] Negotiation successful
0806 16:06:31.595359 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1659,"thread_start_us":150,"threads_started":1}
I20260806 16:06:31.597919 32090 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36291
I20260806 16:06:31.598181 32090 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:31.598995 32090 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:31.600149 31602 ts_manager.cc:194] Registered new tserver with Master: 9913a4a033f24950af4daf47047e75ca (127.13.148.2:37515)
I20260806 16:06:31.600879 31602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:37871
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:06:31.633350 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:42001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36291
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:31.795554 32095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:31.795858 32095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:31.795990 32095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:31.800055 32095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:31.800137 32095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:31.800249 32095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:06:31.806299 32095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.148.20:42001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:31.807751 32095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:31.809211 32095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:31.821949 32104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:31.822314 32102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:31.826566 32095 server_base.cc:1034] running on GCE node
W20260806 16:06:31.831378 32100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:31.832005 32095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:31.832821 32095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:31.840265 32095 hybrid_clock.cc:648] HybridClock initialized: now 1786032391840207 us; error 67 us; skew 500 ppm
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:31.843462 32095 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:06:31.844627 32095 webserver.cc:460] Webserver started at http://127.13.148.3:33927/ using document root <none> and password file <none>
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:31.845351 32095 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:31.845455 32095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:31.845733 32095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:31.862268 32095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "5075a071aac74f48a208915f7b25c5f5"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "159d6f628ba57a3ce9a1301a0eb4e420"
server_key_iv: "37e9e580595b01238f3ebb0107321bdd"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.863432 32095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "5075a071aac74f48a208915f7b25c5f5"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "159d6f628ba57a3ce9a1301a0eb4e420"
server_key_iv: "37e9e580595b01238f3ebb0107321bdd"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.871323 32095 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260806 16:06:31.895076 32117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:31.896927 32095 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260806 16:06:31.897127 32095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "5075a071aac74f48a208915f7b25c5f5"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "159d6f628ba57a3ce9a1301a0eb4e420"
server_key_iv: "37e9e580595b01238f3ebb0107321bdd"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:31.897360 32095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:31.911957 32095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:31.916033 32095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:31.916311 32095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:31.917117 32095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:31.918388 32095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:31.918522 32095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:31.918635 32095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:31.918704 32095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:31.930498 32095 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:43549
I20260806 16:06:31.931807 32095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:06:31.933390 32273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:43549 every 8 connection(s)
I20260806 16:06:31.938858 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 32095
I20260806 16:06:31.938984 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:06:31.939316 13904 external_mini_cluster.cc:1428] Setting key 3fb74548a18f5016c38b1a30249ece0a
Aug 06 16:06:31 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032391, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:31.958915 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:31.937813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:52937 (local address 127.13.148.62:36291)
0806 16:06:31.937956 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:31.937961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:31.943041 (+ 5080us) server_negotiation.cc:407] Connection header received
0806 16:06:31.944377 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:31.944382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:31.944445 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:31.944533 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:31.946149 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.946965 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.948719 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.949046 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.949121 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:31.954269 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:31.954273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:31.954276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:31.954307 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:31.956188 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:31.956892 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:31.956896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:31.956898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:31.957000 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:31.957457 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:31.957460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:31.957462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:31.957696 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:31.957792 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:31.958606 (+ 814us) server_negotiation.cc:299] Negotiation successful
0806 16:06:31.958738 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:06:31.960858 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:31.935823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36291 (local address 127.13.148.3:52937)
0806 16:06:31.942832 (+ 7009us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:31.942906 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:31.944105 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:31.944878 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:31.944888 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:31.945270 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:31.945977 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:31.945997 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.947130 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.947133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:31.948572 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:31.948584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:31.949333 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:31.949337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:31.949549 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:31.950309 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:31.950330 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:31.954048 (+ 3718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:31.956353 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:31.956359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:31.956377 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:31.956772 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:31.957120 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:31.957123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:31.957125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:31.957341 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:31.957886 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:31.957893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:31.958274 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 16:06:31.959408 (+ 1134us) client_negotiation.cc:240] Negotiation successful
0806 16:06:31.959776 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6718,"thread_start_us":151,"threads_started":1}
I20260806 16:06:31.962445 32274 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36291
I20260806 16:06:31.962716 32274 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:31.963335 32274 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:31.966148 31602 ts_manager.cc:194] Registered new tserver with Master: 5075a071aac74f48a208915f7b25c5f5 (127.13.148.3:43549)
I20260806 16:06:31.966952 31602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:52937
I20260806 16:06:31.977782 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:06:31.978590 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 31714
I20260806 16:06:32.009672 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 31905
I20260806 16:06:32.061326 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 32095
I20260806 16:06:32.079291 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 31554
I20260806 16:06:32.091297 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36291
--webserver_interface=127.13.148.62
--webserver_port=35755
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36291
--ranger_config_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:42001/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:42001/kms with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:32.209383 32278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:32.209684 32278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:32.209725 32278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:32.213903 32278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:32.213974 32278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:32.213994 32278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:32.214011 32278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:32.214027 32278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:32.219782 32278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:42001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36291
--ranger_config_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=35755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:32.220849 32278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:32.221879 32278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:32.230758 32287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:32.230751 32286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:32.230903 32289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:32.231288 32278 server_base.cc:1034] running on GCE node
I20260806 16:06:32.231912 32278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:32.233336 32278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:32.234608 32278 hybrid_clock.cc:648] HybridClock initialized: now 1786032392234582 us; error 41 us; skew 500 ppm
Aug 06 16:06:32 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032392, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:32.237870 32278 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:06:32.239084 32278 webserver.cc:460] Webserver started at http://127.13.148.62:35755/ using document root <none> and password file <none>
I20260806 16:06:32.239702 32278 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:32.239811 32278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 16:06:32 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032392, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:32.297479 32278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 16:06:32.300643 32302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:32.302376 32278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:06:32.302563 32278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "ffbee2b3100243cd8441104a101a5d7f"
format_stamp: "Formatted at 2026-08-06 16:06:30 on dist-test-slave-7kzw"
server_key: "88fb95c07845d326ae0b9edc7b36ae29"
server_key_iv: "108692f94df4b7ca72013ad8badb1ed8"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:32.303318 32278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:32.320709 32278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:32.324540 32278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:32.324769 32278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:32.334913 32278 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36291
I20260806 16:06:32.334914 32372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36291 every 8 connection(s)
I20260806 16:06:32.336522 32278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:06:32.340293 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 32278
I20260806 16:06:32.340852 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:42001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:41241
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:36291
--webserver_port=42083
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:42001/kms with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:06:32.341485 32374 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:06:32.343823 32374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Bootstrap starting.
I20260806 16:06:32.367439 32374 log.cc:824] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:32.376267 32374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:06:32.376744 32374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Bootstrap complete.
I20260806 16:06:32.382368 32374 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } }
I20260806 16:06:32.382879 32374 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbee2b3100243cd8441104a101a5d7f, State: Initialized, Role: FOLLOWER
I20260806 16:06:32.383468 32374 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } }
I20260806 16:06:32.383615 32374 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:06:32.383709 32374 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:06:32.383808 32374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:06:32.386242 32374 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } }
I20260806 16:06:32.386621 32374 leader_election.cc:302] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffbee2b3100243cd8441104a101a5d7f; no voters:
I20260806 16:06:32.387085 32374 leader_election.cc:288] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 16:06:32.387312 32379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:06:32.387678 32379 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [term 2 LEADER]: Becoming Leader. State: Replica: ffbee2b3100243cd8441104a101a5d7f, State: Running, Role: LEADER
I20260806 16:06:32.388077 32379 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } }
I20260806 16:06:32.388540 32374 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:06:32.389225 32379 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ffbee2b3100243cd8441104a101a5d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } } }
I20260806 16:06:32.389423 32379 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:32.389700 32379 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffbee2b3100243cd8441104a101a5d7f. Latest consensus state: current_term: 2 leader_uuid: "ffbee2b3100243cd8441104a101a5d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbee2b3100243cd8441104a101a5d7f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36291 } } }
I20260806 16:06:32.389840 32379 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:32.390986 32374 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 16:06:32.391364 32391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:06:32.398944 32391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:06:32.399823 32391 catalog_manager.cc:1259] Loaded cluster ID: c17fe94255924f6cb73232b10be27a69
I20260806 16:06:32.399972 32391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:06:32.403234 32391 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:06:32.404044 32391 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Loaded TSK: 0
I20260806 16:06:32.405411 32391 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:06:32.508129 32377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:32.508499 32377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:32.508566 32377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:32.514782 32377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:32.514870 32377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:32.514961 32377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:06:32.523015 32377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:42001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:41241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=42083
--tserver_master_addrs=127.13.148.62:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:32.524681 32377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:32.526090 32377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:32.538933 32426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:32.539209 32377 server_base.cc:1034] running on GCE node
W20260806 16:06:32.539260 32425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:32.539556 32431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:32.540071 32377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:32.541056 32377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:32.542215 32377 hybrid_clock.cc:648] HybridClock initialized: now 1786032392542149 us; error 80 us; skew 500 ppm
Aug 06 16:06:32 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032392, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:32.546183 32377 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:06:32.547420 32377 webserver.cc:460] Webserver started at http://127.13.148.1:42083/ using document root <none> and password file <none>
I20260806 16:06:32.548208 32377 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:32.548334 32377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 16:06:32 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032392, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:32.576301 32377 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260806 16:06:32.581308 32443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:32.583847 32377 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 16:06:32.584065 32377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "2dbb8570c146410caffc4b5a28a57cc7"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "d92779eb34e772f103de3d746987254b"
server_key_iv: "3261048d2f4a40d64cf21484d2f9e979"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:32.584807 32377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:32.603279 32377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:32.607411 32377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:32.607630 32377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:32.608326 32377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:32.609313 32377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:32.609372 32377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:32.609426 32377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:32.609447 32377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:32.619475 32377 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:41241
I20260806 16:06:32.620563 32377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:06:32.622666 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 32377
I20260806 16:06:32.623167 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:42001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:37515
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:36291
--webserver_port=36421
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:42001/kms with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:06:32.625182 32601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:41241 every 8 connection(s)
Aug 06 16:06:32 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032392, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:32.643972 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:32.624421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:60703 (local address 127.13.148.62:36291)
0806 16:06:32.626391 (+ 1970us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:32.626408 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:32.626443 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:06:32.626820 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:32.626837 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:32.627295 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:32.627654 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:32.629329 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:32.631019 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:32.632970 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:32.633810 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:32.633998 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:32.637571 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:32.637580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:32.637602 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:32.637657 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:32.640044 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:32.640693 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:32.640700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:32.640705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:32.640844 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:32.641224 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:32.641227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:32.641229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:32.641723 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:32.641840 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:32.642666 (+ 826us) server_negotiation.cc:299] Negotiation successful
0806 16:06:32.642957 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"thread_start_us":142,"threads_started":1}
I20260806 16:06:32.645584 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:32.624581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36291 (local address 127.13.148.1:60703)
0806 16:06:32.625111 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:32.625163 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:32.625993 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:32.628164 (+ 2171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:32.628175 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:32.628502 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:32.629132 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:32.629154 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:32.631191 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:32.631197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:32.632804 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:32.632820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:32.634252 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:32.634259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:32.634533 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:32.635342 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:32.635369 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:32.637314 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:32.640186 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:32.640193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:32.640208 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:32.640544 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:32.640947 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:32.640950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:32.640952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:32.641117 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:32.641905 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:32.641912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:32.642198 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 16:06:32.642396 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 16:06:32.644762 (+ 2366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":137,"threads_started":1}
I20260806 16:06:32.649096 32602 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36291
I20260806 16:06:32.649382 32602 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:32.649974 32602 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
W20260806 16:06:32.652580 32602 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:36291 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 16:06:32.652717 32602 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:32.652913 32602 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:32.654016 32602 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:32.654218 32602 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
W20260806 16:06:32.655164 32602 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:06:32.784052 32611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:32.784525 32611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:32.784574 32611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:32.789373 32611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:32.789458 32611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:32.789563 32611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:06:32.795378 32611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:42001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:37515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=36421
--tserver_master_addrs=127.13.148.62:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:32.796494 32611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:32.797544 32611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:32.807816 32618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:32.807901 32622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:32.807816 32620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:32.809281 32611 server_base.cc:1034] running on GCE node
I20260806 16:06:32.809916 32611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:32.810853 32611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:32.812124 32611 hybrid_clock.cc:648] HybridClock initialized: now 1786032392812090 us; error 51 us; skew 500 ppm
Aug 06 16:06:32 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032392, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:32.816090 32611 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:06:32.817278 32611 webserver.cc:460] Webserver started at http://127.13.148.2:36421/ using document root <none> and password file <none>
I20260806 16:06:32.818043 32611 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:32.818154 32611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 16:06:32 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032392, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:32.838943 32611 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:06:32.845544 32632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:32.847357 32611 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260806 16:06:32.847507 32611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "9913a4a033f24950af4daf47047e75ca"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "9080a12ddd37bfe2fa59ed94aee9c325"
server_key_iv: "1fee781a8d456751fbc5c1da5630181a"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:32.848277 32611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:32.863917 32611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:32.879433 32611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:32.879688 32611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:32.880489 32611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:32.881814 32611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:32.881892 32611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:32.881963 32611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:32.881999 32611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:32.897930 32611 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:37515
I20260806 16:06:32.898049 324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:37515 every 8 connection(s)
I20260806 16:06:32.899339 32611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:06:32.906982 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 32611
I20260806 16:06:32.907523 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:42001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:43549
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:36291
--webserver_port=33927
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:42001/kms with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:06:32 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032392, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:32.922510 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:32.904587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:46611 (local address 127.13.148.62:36291)
0806 16:06:32.904778 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:32.904783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:32.905557 (+ 774us) server_negotiation.cc:407] Connection header received
0806 16:06:32.906746 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:32.906750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:32.906820 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:32.906944 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:32.908445 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:32.909397 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:32.911438 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:32.911841 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:32.911964 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:32.915443 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:32.915447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:32.915451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:32.915483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:32.919003 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:32.920277 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:32.920282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:32.920283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:32.920424 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:32.920896 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:32.920900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:32.920902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:32.921155 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:32.921270 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:32.922024 (+ 754us) server_negotiation.cc:299] Negotiation successful
0806 16:06:32.922230 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 16:06:32.924386 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:32.904841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36291 (local address 127.13.148.2:46611)
0806 16:06:32.905396 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:32.905451 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:32.906458 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:32.907149 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:32.907158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:32.907541 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:32.908258 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:32.908281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:32.909549 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:32.909556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:32.911121 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:32.911138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:32.912027 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:32.912034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:32.912439 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:32.913221 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:32.913248 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:32.915245 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:32.919649 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:32.919659 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:32.919684 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:32.920140 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:32.920527 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:32.920533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:32.920537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:32.920776 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:32.921270 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:32.921277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:32.921671 (+ 394us) client_negotiation.cc:769] Sending connection context
0806 16:06:32.922788 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0806 16:06:32.923218 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":174,"threads_started":1}
I20260806 16:06:32.925913 325 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36291
I20260806 16:06:32.926218 325 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:32.926837 325 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
W20260806 16:06:32.928733 325 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:36291 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 16:06:32.928870 325 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:32.929077 325 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:32.932423 325 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:32.932658 325 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
W20260806 16:06:32.933576 325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:06:33.057642 328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:33.058087 328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:33.058255 328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:33.064386 328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:33.064538 328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:33.064728 328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:06:33.070847 328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:42001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:43549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=33927
--tserver_master_addrs=127.13.148.62:36291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:33.071940 328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:33.073010 328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:33.087590 338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:33.087597 340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:33.088120 343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:33.088502 328 server_base.cc:1034] running on GCE node
I20260806 16:06:33.089016 328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:33.090737 328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:33.091956 328 hybrid_clock.cc:648] HybridClock initialized: now 1786032393091920 us; error 50 us; skew 500 ppm
Aug 06 16:06:33 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032393, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:33.095487 328 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:06:33.096468 328 webserver.cc:460] Webserver started at http://127.13.148.3:33927/ using document root <none> and password file <none>
I20260806 16:06:33.097062 328 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:33.097142 328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 16:06:33 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032393, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:33.128177 328 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.001s
I20260806 16:06:33.131448 351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:33.133041 328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:06:33.133172 328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "5075a071aac74f48a208915f7b25c5f5"
format_stamp: "Formatted at 2026-08-06 16:06:31 on dist-test-slave-7kzw"
server_key: "159d6f628ba57a3ce9a1301a0eb4e420"
server_key_iv: "37e9e580595b01238f3ebb0107321bdd"
server_key_version: "kuduclusterkey@0"
I20260806 16:06:33.133922 328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:33.154765 328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:33.158200 328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:33.158428 328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:33.159204 328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:33.160280 328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:33.160353 328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:33.160414 328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:33.160449 328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:33.173976 328 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:43549
I20260806 16:06:33.174026 503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:43549 every 8 connection(s)
I20260806 16:06:33.175362 328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:06:33.182010 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 328
Aug 06 16:06:33 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032393, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:33.195474 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:33.179983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:36697 (local address 127.13.148.62:36291)
0806 16:06:33.180123 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:33.180128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:33.181030 (+ 902us) server_negotiation.cc:407] Connection header received
0806 16:06:33.182044 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:33.182047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:33.182119 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:33.182223 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:33.183565 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:33.184369 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:33.185833 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:33.186168 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:33.186252 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:33.191582 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:33.191587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:33.191590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:33.191619 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:33.193304 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:33.193850 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:33.193854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:33.193856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:33.193942 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:33.194302 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:33.194306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:33.194307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:33.194550 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:33.194653 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:33.195162 (+ 509us) server_negotiation.cc:299] Negotiation successful
0806 16:06:33.195298 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 16:06:33.196431 506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:33.180421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36291 (local address 127.13.148.3:36697)
0806 16:06:33.180871 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:33.180925 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:33.181834 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:33.182414 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:33.182439 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:33.182852 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:33.183402 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:33.183416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:33.184549 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:33.184553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:33.185685 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:33.185698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:33.186487 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:33.186492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:33.186698 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:33.187354 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:33.187377 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:33.191380 (+ 4003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:33.193431 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:33.193436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:33.193452 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:33.193743 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:33.194036 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:33.194040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:33.194042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:33.194196 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:33.194664 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:33.194672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:33.194919 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 16:06:33.195113 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 16:06:33.195424 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":131,"threads_started":1}
I20260806 16:06:33.197890 504 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36291
I20260806 16:06:33.198151 504 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:33.198786 504 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
W20260806 16:06:33.200685 504 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:36291 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 16:06:33.200837 504 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:33.201040 504 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:33.201987 504 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:33.202186 504 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:33.202450 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:33.182798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.13.148.62:36291)
0806 16:06:33.183516 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:33.183521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:33.183535 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:06:33.183576 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:33.183579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:33.183630 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:33.183731 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:33.184847 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:33.185708 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:33.195687 (+ 9979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:33.196057 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:33.196194 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:33.198598 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:33.198603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:33.198605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:33.198631 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:33.200394 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:33.200929 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:33.200932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:33.200934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:33.201024 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:33.201387 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:33.201390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:33.201392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:33.201605 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:33.201837 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:33.201988 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 16:06:33.202116 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":103,"threads_started":1}
W20260806 16:06:33.203255 504 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 16:06:33 dist-test-slave-7kzw krb5kdc[30934](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786032393, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:33.655414 32602 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:33.655742 32602 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:33.657043 32326 ts_manager.cc:194] Registered new tserver with Master: 2dbb8570c146410caffc4b5a28a57cc7 (127.13.148.1:41241)
I20260806 16:06:33.658833 32326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:60703
I20260806 16:06:33.933851 325 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:33.934211 325 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:33.935205 32325 ts_manager.cc:194] Registered new tserver with Master: 9913a4a033f24950af4daf47047e75ca (127.13.148.2:37515)
I20260806 16:06:33.936095 32325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:46611
Aug 06 16:06:34 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032393, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:34.203511 504 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:34.203837 504 heartbeater.cc:511] Master 127.13.148.62:36291 requested a full tablet report, sending...
I20260806 16:06:34.204584 32325 ts_manager.cc:194] Registered new tserver with Master: 5075a071aac74f48a208915f7b25c5f5 (127.13.148.3:43549)
I20260806 16:06:34.205341 32325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:36697
I20260806 16:06:34.216810 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:06:34.228236 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.218830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.13.148.62:36291)
0806 16:06:34.219174 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:34.219179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:34.219195 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:06:34.219288 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:34.219291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:34.219345 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:34.219456 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:34.220384 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.221241 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.222400 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.222802 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.222913 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.224496 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:34.224500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:34.224503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:34.224532 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:34.226353 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:34.226858 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:34.226864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:34.226867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:34.226970 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:34.227284 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:34.227288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:34.227290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:34.227527 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:34.227684 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:34.227909 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 16:06:34.228047 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":167,"threads_started":1}
I20260806 16:06:34.232607 32325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:06:34.237517 32325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:06:34.264201 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.255653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37515 (local address 127.0.0.1:59196)
0806 16:06:34.257789 (+ 2136us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:34.257842 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:34.257972 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:34.259174 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:34.259181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:34.259206 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:34.259348 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.259354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.261236 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.261240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.262577 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.262588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.263687 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.263691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.263803 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.263849 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 16:06:34.263933 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 16:06:34.264049 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1930,"thread_start_us":123,"threads_started":1}
I20260806 16:06:34.265453 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.256589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.13.148.2:37515)
0806 16:06:34.258671 (+ 2082us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:34.258678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:34.258703 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:06:34.258780 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:34.258785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:34.258990 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:34.259498 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:34.259638 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.261113 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.262737 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.263537 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.265025 (+ 1488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.265178 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:34.265227 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 16:06:34.265318 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1960,"thread_start_us":98,"threads_started":1}
I20260806 16:06:34.268642 32700 tablet_service.cc:1467] Processing CreateTablet for tablet a31137a7b2b54ddd9a88fc6629afc6f8 (DEFAULT_TABLE table=test-table [id=2cdb24dda7b640368d53f10b82c8c049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:34.269768 32700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31137a7b2b54ddd9a88fc6629afc6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:34.271549 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.255699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:43549 (local address 127.0.0.1:41452)
0806 16:06:34.264825 (+ 9126us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:34.264838 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:34.264904 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:34.267446 (+ 2542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:34.267449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:34.267468 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:34.267544 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.267552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.268900 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.268903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.269947 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.269958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.271142 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.271146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.271240 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.271266 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 16:06:34.271322 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 16:06:34.271415 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9070,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260806 16:06:34.272279 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.256399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.13.148.3:43549)
0806 16:06:34.266911 (+ 10512us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:34.266918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:34.266945 (+ 27us) server_negotiation.cc:407] Connection header received
0806 16:06:34.267027 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:34.267033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:34.267264 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:34.267460 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:34.267649 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.268772 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.270091 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.270999 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.271847 (+ 848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.271991 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:34.272042 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 16:06:34.272137 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10402,"thread_start_us":202,"threads_started":1}
I20260806 16:06:34.273609 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.257302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:41241 (local address 127.0.0.1:58720)
0806 16:06:34.267469 (+ 10167us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:34.267484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:34.267573 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:34.269516 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:34.269520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:34.269537 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:34.269622 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.269628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.270991 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.270995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.272149 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.272172 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.273245 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.273250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.273336 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.273360 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:06:34.273408 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 16:06:34.273472 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10101,"mutex_wait_us":5,"thread_start_us":175,"threads_started":1}
I20260806 16:06:34.274334 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.257813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.13.148.1:41241)
0806 16:06:34.266684 (+ 8871us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:34.266691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:34.268929 (+ 2238us) server_negotiation.cc:407] Connection header received
0806 16:06:34.269027 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:34.269033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:34.269224 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:34.269366 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:34.269756 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.270853 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.272327 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.273088 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.273886 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.274031 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:34.274085 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 16:06:34.274187 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8749,"thread_start_us":74,"threads_started":1}
I20260806 16:06:34.276010 417 tablet_service.cc:1467] Processing CreateTablet for tablet a31137a7b2b54ddd9a88fc6629afc6f8 (DEFAULT_TABLE table=test-table [id=2cdb24dda7b640368d53f10b82c8c049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:34.277087 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31137a7b2b54ddd9a88fc6629afc6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:34.277005 32510 tablet_service.cc:1467] Processing CreateTablet for tablet a31137a7b2b54ddd9a88fc6629afc6f8 (DEFAULT_TABLE table=test-table [id=2cdb24dda7b640368d53f10b82c8c049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:34.278056 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a31137a7b2b54ddd9a88fc6629afc6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:34.287919 554 tablet_bootstrap.cc:492] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Bootstrap starting.
I20260806 16:06:34.290683 554 tablet_bootstrap.cc:654] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:34.291710 554 log.cc:824] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:34.292486 556 tablet_bootstrap.cc:492] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7: Bootstrap starting.
I20260806 16:06:34.296517 554 tablet_bootstrap.cc:492] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: No bootstrap required, opened a new log
I20260806 16:06:34.296828 554 ts_tablet_manager.cc:1397] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260806 16:06:34.297787 556 tablet_bootstrap.cc:654] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:34.298704 556 log.cc:824] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:34.300752 555 tablet_bootstrap.cc:492] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Bootstrap starting.
I20260806 16:06:34.301393 554 raft_consensus.cc:348] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.301808 554 raft_consensus.cc:374] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:34.301913 554 raft_consensus.cc:729] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9913a4a033f24950af4daf47047e75ca, State: Initialized, Role: FOLLOWER
I20260806 16:06:34.302529 554 consensus_queue.cc:260] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.303735 554 ts_tablet_manager.cc:1428] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:06:34.304006 555 tablet_bootstrap.cc:654] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:34.304271 325 heartbeater.cc:503] Master 127.13.148.62:36291 was elected leader, sending a full tablet report...
I20260806 16:06:34.304867 555 log.cc:824] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:34.308354 556 tablet_bootstrap.cc:492] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7: No bootstrap required, opened a new log
I20260806 16:06:34.308636 556 ts_tablet_manager.cc:1397] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
Aug 06 16:06:34 dist-test-slave-7kzw krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032393, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260806 16:06:34.313050 556 raft_consensus.cc:348] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.313439 556 raft_consensus.cc:374] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:34.313540 556 raft_consensus.cc:729] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dbb8570c146410caffc4b5a28a57cc7, State: Initialized, Role: FOLLOWER
I20260806 16:06:34.314138 556 consensus_queue.cc:260] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.315460 556 ts_tablet_manager.cc:1428] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 16:06:34.315917 32602 heartbeater.cc:503] Master 127.13.148.62:36291 was elected leader, sending a full tablet report...
I20260806 16:06:34.319542 555 tablet_bootstrap.cc:492] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: No bootstrap required, opened a new log
I20260806 16:06:34.319862 555 ts_tablet_manager.cc:1397] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260806 16:06:34.324147 555 raft_consensus.cc:348] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.324533 555 raft_consensus.cc:374] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:34.324642 555 raft_consensus.cc:729] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5075a071aac74f48a208915f7b25c5f5, State: Initialized, Role: FOLLOWER
I20260806 16:06:34.325253 555 consensus_queue.cc:260] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.326627 555 ts_tablet_manager.cc:1428] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 16:06:34.326957 504 heartbeater.cc:503] Master 127.13.148.62:36291 was elected leader, sending a full tablet report...
W20260806 16:06:34.375434 32605 tablet.cc:2369] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:06:34.401769 326 tablet.cc:2369] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:06:34.428511 505 tablet.cc:2369] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:06:34.542048 562 raft_consensus.cc:482] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:34.542359 562 raft_consensus.cc:504] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.543877 562 leader_election.cc:288] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9913a4a033f24950af4daf47047e75ca (127.13.148.2:37515), 2dbb8570c146410caffc4b5a28a57cc7 (127.13.148.1:41241)
I20260806 16:06:34.549787 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.544999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:45459 (local address 127.13.148.2:37515)
0806 16:06:34.545154 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:34.545159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:34.545345 (+ 186us) server_negotiation.cc:407] Connection header received
0806 16:06:34.545485 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:34.545489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:34.545550 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:34.545652 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:34.546154 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.546906 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.548447 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.549026 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.549132 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.549203 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:34.549563 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 16:06:34.549644 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:06:34.550134 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.544787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:47877 (local address 127.13.148.1:41241)
0806 16:06:34.544924 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:34.544928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:34.545692 (+ 764us) server_negotiation.cc:407] Connection header received
0806 16:06:34.545880 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:34.545883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:34.545928 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:34.545973 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:34.546406 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.547258 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.548771 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.549441 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.549536 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.549583 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:34.549924 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 16:06:34.550007 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:06:34.550571 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.544611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:41241 (local address 127.13.148.3:47877)
0806 16:06:34.545608 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:34.545620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:34.545761 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:34.546108 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:34.546111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:34.546128 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:34.546266 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.546272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.547420 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.547424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.548616 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.548628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.549679 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.549682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.549791 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.549812 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:06:34.550299 (+ 487us) client_negotiation.cc:240] Negotiation successful
0806 16:06:34.550380 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":102,"threads_started":1}
I20260806 16:06:34.551340 32522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31137a7b2b54ddd9a88fc6629afc6f8" candidate_uuid: "5075a071aac74f48a208915f7b25c5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dbb8570c146410caffc4b5a28a57cc7" is_pre_election: true
I20260806 16:06:34.551730 32522 raft_consensus.cc:2454] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5075a071aac74f48a208915f7b25c5f5 in term 0.
I20260806 16:06:34.552222 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.544177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37515 (local address 127.13.148.3:45459)
0806 16:06:34.545235 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:34.545252 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:34.545380 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:34.545815 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:34.545819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:34.545841 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:34.546001 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.546008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.547062 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.547065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.548290 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:34.548301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.549308 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.549311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:34.549411 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.549435 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:06:34.550631 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0806 16:06:34.550796 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":128,"threads_started":1}
I20260806 16:06:34.552306 353 leader_election.cc:302] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dbb8570c146410caffc4b5a28a57cc7, 5075a071aac74f48a208915f7b25c5f5; no voters:
I20260806 16:06:34.552600 562 raft_consensus.cc:2781] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:06:34.552702 562 raft_consensus.cc:482] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:06:34.552754 562 raft_consensus.cc:3037] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:34.553170 32731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31137a7b2b54ddd9a88fc6629afc6f8" candidate_uuid: "5075a071aac74f48a208915f7b25c5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9913a4a033f24950af4daf47047e75ca" is_pre_election: true
I20260806 16:06:34.553452 32731 raft_consensus.cc:2454] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5075a071aac74f48a208915f7b25c5f5 in term 0.
I20260806 16:06:34.554219 562 raft_consensus.cc:504] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.554641 562 leader_election.cc:288] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [CANDIDATE]: Term 1 election: Requested vote from peers 9913a4a033f24950af4daf47047e75ca (127.13.148.2:37515), 2dbb8570c146410caffc4b5a28a57cc7 (127.13.148.1:41241)
I20260806 16:06:34.555043 32731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31137a7b2b54ddd9a88fc6629afc6f8" candidate_uuid: "5075a071aac74f48a208915f7b25c5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9913a4a033f24950af4daf47047e75ca"
I20260806 16:06:34.555056 32522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a31137a7b2b54ddd9a88fc6629afc6f8" candidate_uuid: "5075a071aac74f48a208915f7b25c5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dbb8570c146410caffc4b5a28a57cc7"
I20260806 16:06:34.555182 32731 raft_consensus.cc:3037] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:34.555197 32522 raft_consensus.cc:3037] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:34.556247 32522 raft_consensus.cc:2454] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5075a071aac74f48a208915f7b25c5f5 in term 1.
I20260806 16:06:34.556533 32731 raft_consensus.cc:2454] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5075a071aac74f48a208915f7b25c5f5 in term 1.
I20260806 16:06:34.556545 353 leader_election.cc:302] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dbb8570c146410caffc4b5a28a57cc7, 5075a071aac74f48a208915f7b25c5f5; no voters:
I20260806 16:06:34.556730 562 raft_consensus.cc:2781] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:06:34.556941 562 raft_consensus.cc:686] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 1 LEADER]: Becoming Leader. State: Replica: 5075a071aac74f48a208915f7b25c5f5, State: Running, Role: LEADER
I20260806 16:06:34.557268 562 consensus_queue.cc:237] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } }
I20260806 16:06:34.560644 32324 catalog_manager.cc:5495] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5075a071aac74f48a208915f7b25c5f5 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "5075a071aac74f48a208915f7b25c5f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5075a071aac74f48a208915f7b25c5f5" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 43549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 } health_report { overall_health: UNKNOWN } } }
I20260806 16:06:34.627526 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.623545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.13.148.3:43549)
0806 16:06:34.623673 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:34.623677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:34.623690 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:06:34.623859 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:34.623861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:34.623917 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:34.624007 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:06:34.624239 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.624725 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.625979 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:34.626183 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:34.626284 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:34.626620 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:06:34.627066 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:06:34.627161 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:34.627314 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 16:06:34.627375 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:06:34.633566 32731 raft_consensus.cc:1261] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 1 FOLLOWER]: Refusing update from remote peer 5075a071aac74f48a208915f7b25c5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:06:34.633566 32522 raft_consensus.cc:1261] T a31137a7b2b54ddd9a88fc6629afc6f8 P 2dbb8570c146410caffc4b5a28a57cc7 [term 1 FOLLOWER]: Refusing update from remote peer 5075a071aac74f48a208915f7b25c5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:06:34.634334 562 consensus_queue.cc:1035] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9913a4a033f24950af4daf47047e75ca" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:06:34.634598 577 consensus_queue.cc:1035] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dbb8570c146410caffc4b5a28a57cc7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:06:34.652326 32324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:06:34.652562 32324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:06:34.655411 32324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffbee2b3100243cd8441104a101a5d7f: Sending DeleteTablet for 3 replicas of tablet a31137a7b2b54ddd9a88fc6629afc6f8
I20260806 16:06:34.656231 417 tablet_service.cc:1514] Processing DeleteTablet for tablet a31137a7b2b54ddd9a88fc6629afc6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:06:34 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:41452
I20260806 16:06:34.656304 32700 tablet_service.cc:1514] Processing DeleteTablet for tablet a31137a7b2b54ddd9a88fc6629afc6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:06:34 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:59196
I20260806 16:06:34.656854 597 tablet_replica.cc:330] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: stopping tablet replica
I20260806 16:06:34.656939 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 32377
I20260806 16:06:34.656947 32510 tablet_service.cc:1514] Processing DeleteTablet for tablet a31137a7b2b54ddd9a88fc6629afc6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:06:34 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:58720
I20260806 16:06:34.657243 597 raft_consensus.cc:2229] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:06:34.657701 597 raft_consensus.cc:2258] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:06:34.661729 597 ts_tablet_manager.cc:1905] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:06:34.664897 597 ts_tablet_manager.cc:1918] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:06:34.664996 597 log.cc:1196] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/a31137a7b2b54ddd9a88fc6629afc6f8
I20260806 16:06:34.665340 597 ts_tablet_manager.cc:1939] T a31137a7b2b54ddd9a88fc6629afc6f8 P 5075a071aac74f48a208915f7b25c5f5: Deleting consensus metadata
I20260806 16:06:34.666687 598 tablet_replica.cc:330] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: stopping tablet replica
I20260806 16:06:34.666764 32308 catalog_manager.cc:4841] TS 5075a071aac74f48a208915f7b25c5f5 (127.13.148.3:43549): tablet a31137a7b2b54ddd9a88fc6629afc6f8 (table test-table [id=2cdb24dda7b640368d53f10b82c8c049]) successfully deleted
I20260806 16:06:34.667047 598 raft_consensus.cc:2229] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:06:34.667397 598 raft_consensus.cc:2258] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:06:34.668955 598 ts_tablet_manager.cc:1905] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:06:34.671811 598 ts_tablet_manager.cc:1918] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:06:34.671954 598 log.cc:1196] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/a31137a7b2b54ddd9a88fc6629afc6f8
I20260806 16:06:34.672329 598 ts_tablet_manager.cc:1939] T a31137a7b2b54ddd9a88fc6629afc6f8 P 9913a4a033f24950af4daf47047e75ca: Deleting consensus metadata
I20260806 16:06:34.673632 32305 catalog_manager.cc:4841] TS 9913a4a033f24950af4daf47047e75ca (127.13.148.2:37515): tablet a31137a7b2b54ddd9a88fc6629afc6f8 (table test-table [id=2cdb24dda7b640368d53f10b82c8c049]) successfully deleted
W20260806 16:06:34.674002 32305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:41241 (error 108)
I20260806 16:06:34.674153 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 32611
I20260806 16:06:34.677659 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:34.677303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:41241 (local address 127.0.0.1:58732)
0806 16:06:34.677463 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:34.677548 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:41241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 16:06:34.677940 32305 catalog_manager.cc:4584] TS 2dbb8570c146410caffc4b5a28a57cc7 (127.13.148.1:41241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a31137a7b2b54ddd9a88fc6629afc6f8: Network error: Client connection negotiation failed: client connection to 127.13.148.1:41241: connect: Connection refused (error 111)
I20260806 16:06:34.686115 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 328
I20260806 16:06:34.696640 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 32278
I20260806 16:06:34.716792 13904 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 16:06:35.062247 13904 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 16:06:35.062408 13904 mini_ranger.cc:66] Stopping Ranger...
I20260806 16:06:35.408102 13904 mini_ranger.cc:68] Stopped Ranger
2026-08-06 16:06:35.408 UTC [30972] LOG: received smart shutdown request
2026-08-06 16:06:35.412 UTC [30972] LOG: background worker "logical replication launcher" (PID 30983) exited with exit code 1
2026-08-06 16:06:35.413 UTC [30977] LOG: shutting down
2026-08-06 16:06:35.588 UTC [30972] LOG: database system is shut down
2026-08-06T16:06:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51620 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:06:35 dist-test-slave-7kzw krb5kdc[614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:06:35 dist-test-slave-7kzw krb5kdc[614](info): set up 2 sockets
Aug 06 16:06:35 dist-test-slave-7kzw krb5kdc[614](info): commencing operation
krb5kdc: starting...
W20260806 16:06:36.933508 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:06:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:06:37.073789 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46491
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46491,127.13.148.61:41905,127.13.148.60:33583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:37.188014 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:37.188271 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:37.188309 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:37.192255 632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:37.192309 632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:37.192329 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:37.192345 632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:37.192412 632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:37.197525 632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46491,127.13.148.61:41905,127.13.148.60:33583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:37.198575 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:37.199594 632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:06:37.208542 632 server_base.cc:1034] running on GCE node
W20260806 16:06:37.208619 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:37.208578 642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:37.208570 644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:37.209692 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:37.210984 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:37.212184 632 hybrid_clock.cc:648] HybridClock initialized: now 1786032397212157 us; error 48 us; skew 500 ppm
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:37.215412 632 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:06:37.216320 632 webserver.cc:460] Webserver started at http://127.13.148.62:36811/ using document root <none> and password file <none>
I20260806 16:06:37.216928 632 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:37.217120 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:37.217412 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:06:37.219830 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "5331a6d261c74fbaa0b425aaac4fe5e3"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "58380bd1e58ae2afb148538d462faaaa"
server_key_iv: "6ebc04704b16c6f7d3fa5df13f02a2d4"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.220522 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "5331a6d261c74fbaa0b425aaac4fe5e3"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "58380bd1e58ae2afb148538d462faaaa"
server_key_iv: "6ebc04704b16c6f7d3fa5df13f02a2d4"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.225178 632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:06:37.227931 665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:37.229174 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:06:37.229367 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "5331a6d261c74fbaa0b425aaac4fe5e3"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "58380bd1e58ae2afb148538d462faaaa"
server_key_iv: "6ebc04704b16c6f7d3fa5df13f02a2d4"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.229486 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:37.261196 632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:37.264652 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:37.264884 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:37.272945 632 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46491
I20260806 16:06:37.272972 733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46491 every 8 connection(s)
I20260806 16:06:37.274020 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:06:37.276994 734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:37.281015 734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:06:37.282671 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 632
I20260806 16:06:37.282835 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:06:37.283203 13904 external_mini_cluster.cc:1428] Setting key 721221fbcfa0c8859b6279a76c058080
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.61@KRBTEST.COM" created.
I20260806 16:06:37.299386 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.284361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.13.148.62:46491)
0806 16:06:37.284633 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.284644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.284686 (+ 42us) server_negotiation.cc:407] Connection header received
0806 16:06:37.285981 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.285991 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.286280 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.286530 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.287403 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.289006 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.290379 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.290860 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.290992 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.293941 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.293946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.293956 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.293978 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.295890 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.296502 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.296507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.296508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.296618 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.296986 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.296992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.296994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.297412 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.297562 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.297945 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.298210 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260806 16:06:37.299386 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.283595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46491 (local address 127.0.0.1:59396)
0806 16:06:37.284170 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.284250 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.285620 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.286559 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.286565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.286943 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.287223 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.287238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.289165 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.289169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.290236 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.290253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.291039 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.291047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.291231 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.291927 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.291957 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.293777 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.296031 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.296040 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.296051 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.296356 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.296722 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.296727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.296729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.296875 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.297533 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.297540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.297696 (+ 156us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.297878 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.298274 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":114,"thread_start_us":121,"threads_started":1}
I20260806 16:06:37.301801 734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:06:37.302592 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.302284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55472)
0806 16:06:37.302407 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.302513 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 16:06:37.302822 666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
I20260806 16:06:37.303727 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.303490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55482)
0806 16:06:37.303611 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.303666 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 16:06:37.303900 734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:41905: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
I20260806 16:06:37.314544 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:41905
--webserver_interface=127.13.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46491,127.13.148.61:41905,127.13.148.60:33583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:06:37.353139 734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } attempt: 1
I20260806 16:06:37.354128 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.353737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55494)
0806 16:06:37.353927 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.354013 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 16:06:37.354966 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.354735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55496)
0806 16:06:37.354844 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.354902 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:06:37.355108 734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:41905: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
I20260806 16:06:37.423290 734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } attempt: 2
I20260806 16:06:37.424261 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.423899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55512)
0806 16:06:37.424084 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.424160 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 16:06:37.425096 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.424850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55518)
0806 16:06:37.424965 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.425020 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 16:06:37.425289 734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:41905: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
W20260806 16:06:37.430948 743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:37.431200 743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:37.431272 743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:37.435420 743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:37.435473 743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:37.435493 743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:37.435554 743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:37.435606 743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:37.440999 743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46491,127.13.148.61:41905,127.13.148.60:33583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:41905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:37.442041 743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:37.443137 743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:37.451606 752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:37.451596 748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:37.451855 743 server_base.cc:1034] running on GCE node
W20260806 16:06:37.451601 749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:37.452476 743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:37.453485 743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:37.454669 743 hybrid_clock.cc:648] HybridClock initialized: now 1786032397454648 us; error 25 us; skew 500 ppm
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:37.457657 743 init.cc:377] Logged in from keytab as kudu/127.13.148.61@KRBTEST.COM (short username kudu)
I20260806 16:06:37.458675 743 webserver.cc:460] Webserver started at http://127.13.148.61:39031/ using document root <none> and password file <none>
I20260806 16:06:37.459254 743 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:37.459317 743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:37.459532 743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:06:37.461812 743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/data/instance:
uuid: "2d732fb5382a4b68aabae6f5db9d365c"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "f3f5f07c10ff06dab2ae0432e59350df"
server_key_iv: "82c4ea4305e10860fe4066492fe5f83e"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.462455 743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/wal/instance:
uuid: "2d732fb5382a4b68aabae6f5db9d365c"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "f3f5f07c10ff06dab2ae0432e59350df"
server_key_iv: "82c4ea4305e10860fe4066492fe5f83e"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.466316 743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:06:37.468971 759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:37.470376 743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:06:37.470570 743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/wal
uuid: "2d732fb5382a4b68aabae6f5db9d365c"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "f3f5f07c10ff06dab2ae0432e59350df"
server_key_iv: "82c4ea4305e10860fe4066492fe5f83e"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.470702 743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:37.506562 743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:37.510057 743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:37.510318 743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:37.516484 734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } attempt: 3
I20260806 16:06:37.517491 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.517096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55528)
0806 16:06:37.517293 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.517382 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 16:06:37.518329 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.518070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55544)
0806 16:06:37.518200 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.518260 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 16:06:37.518479 734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:41905: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41905: connect: Connection refused (error 111)
I20260806 16:06:37.518668 743 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:41905
I20260806 16:06:37.518837 834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:41905 every 8 connection(s)
I20260806 16:06:37.519766 743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
I20260806 16:06:37.522141 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 743
I20260806 16:06:37.522303 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-1/wal/instance
I20260806 16:06:37.522681 13904 external_mini_cluster.cc:1428] Setting key d9dfda563ad52cf098842e18cfb97af5
I20260806 16:06:37.522984 835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:37.527842 835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.148.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.60@KRBTEST.COM" created.
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:37.543642 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.530169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.13.148.62:46491)
0806 16:06:37.530369 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.530377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.531302 (+ 925us) server_negotiation.cc:407] Connection header received
0806 16:06:37.532278 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.532285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.532361 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.532475 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.533820 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.534832 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.536127 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.536445 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.536532 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.539402 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.539406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.539408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.539433 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.541220 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.541759 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.541765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.541768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.541875 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.542263 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.542269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.542274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.542568 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.542675 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.543263 (+ 588us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.543415 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 16:06:37.544356 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.530608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46491 (local address 127.0.0.1:59408)
0806 16:06:37.531123 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.531178 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.532022 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.532613 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.532619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.533099 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.533650 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.533673 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.535052 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.535054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.535987 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.535998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.536610 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.536613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.536822 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.537446 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.537463 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.539228 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.541357 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.541361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.541373 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.541646 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.542004 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.542006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.542008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.542151 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.542733 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.542738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.542974 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.543139 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.543394 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":137,"threads_started":1}
I20260806 16:06:37.545902 835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:06:37.555994 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.546425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55548)
0806 16:06:37.546548 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.546562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.546636 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.547290 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.547292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.547469 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.547537 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.547542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.548509 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.548512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.549017 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.549024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.549698 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.549701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.549813 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.550304 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.550316 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.551891 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.554176 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.554179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.554181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.554382 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.554858 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.554860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.554861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.554946 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.555505 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.555508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.555622 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.555763 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.555871 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 16:06:37.556218 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.546611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.13.148.61:41905)
0806 16:06:37.546948 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.546964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.546988 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:06:37.547020 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.547024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.547165 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.547314 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.547642 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.548373 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.549128 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.549566 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.549685 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.552046 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.552055 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.552096 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.552125 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.554060 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.554585 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.554590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.554592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.554750 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.555075 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.555081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.555085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.555378 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.555502 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.555854 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.555974 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":172,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
I20260806 16:06:37.557804 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:33583
--webserver_interface=127.13.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46491,127.13.148.61:41905,127.13.148.60:33583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:06:37.558074 835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:06:37.558835 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.558567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:42992)
0806 16:06:37.558707 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.558777 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:06:37.559095 767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
I20260806 16:06:37.560074 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.559866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43000)
0806 16:06:37.559972 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.560018 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 16:06:37.560232 835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:33583: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
I20260806 16:06:37.609490 835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } attempt: 1
I20260806 16:06:37.610517 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.610155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43008)
0806 16:06:37.610315 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.610383 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 16:06:37.611280 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.611062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43010)
0806 16:06:37.611172 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.611220 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:06:37.611433 835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:33583: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
I20260806 16:06:37.661654 734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } attempt: 4
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
W20260806 16:06:37.678434 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:37.678782 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:37.678874 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:06:37.679600 835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } attempt: 2
I20260806 16:06:37.680306 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.679995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43020)
0806 16:06:37.680123 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.680178 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 16:06:37.681258 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.681055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43026)
0806 16:06:37.681138 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.681183 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 16:06:37.681288 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.662312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55554)
0806 16:06:37.662601 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.662618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.662713 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.670683 (+ 7970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.670686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.670920 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.671088 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.671097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.672797 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.672802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.674266 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.674279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.675012 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.675016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.675164 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.675787 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.675803 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.677407 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.679220 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.679224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.679226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.679437 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.679840 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.679843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.679845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.679951 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.680763 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.680766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.680877 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.680976 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.681115 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
W20260806 16:06:37.681375 835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:33583: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
I20260806 16:06:37.681602 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.663067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.13.148.61:41905)
0806 16:06:37.670324 (+ 7257us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.670330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.670350 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:06:37.670403 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.670407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.670488 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.671232 (+ 744us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.671373 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.672647 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.674447 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.674842 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.674992 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.677549 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.677552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.677554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.677579 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.679114 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.679548 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.679551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.679552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.679659 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.680443 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.680448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.680450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.680656 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.680729 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.681057 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.681520 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7124}
I20260806 16:06:37.682283 734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:06:37.682978 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.682767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43032)
0806 16:06:37.682877 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.682924 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 16:06:37.683576 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.683396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43046)
0806 16:06:37.683486 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.683528 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 16:06:37.683740 734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:33583: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
W20260806 16:06:37.684245 847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:37.684322 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:37.684367 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:37.684404 847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:37.684440 847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:37.689934 847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46491,127.13.148.61:41905,127.13.148.60:33583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:33583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:37.691115 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:37.692130 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:37.700675 855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:37.700675 858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:37.700834 856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:37.701062 847 server_base.cc:1034] running on GCE node
I20260806 16:06:37.701654 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:37.702724 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:37.703902 847 hybrid_clock.cc:648] HybridClock initialized: now 1786032397703874 us; error 37 us; skew 500 ppm
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:37.706771 847 init.cc:377] Logged in from keytab as kudu/127.13.148.60@KRBTEST.COM (short username kudu)
I20260806 16:06:37.707700 847 webserver.cc:460] Webserver started at http://127.13.148.60:34547/ using document root <none> and password file <none>
I20260806 16:06:37.708257 847 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:37.708346 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:37.708576 847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:06:37.710647 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/data/instance:
uuid: "1a41a2bb731c4984842b8e05a41c6d4a"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "b9537741703b9d307da2eb5295f726fe"
server_key_iv: "a1ea5a37567bea422011f98d33176ecd"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.711252 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/wal/instance:
uuid: "1a41a2bb731c4984842b8e05a41c6d4a"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "b9537741703b9d307da2eb5295f726fe"
server_key_iv: "a1ea5a37567bea422011f98d33176ecd"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.715309 847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:06:37.717978 866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:37.719281 847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:06:37.719429 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/wal
uuid: "1a41a2bb731c4984842b8e05a41c6d4a"
format_stamp: "Formatted at 2026-08-06 16:06:37 on dist-test-slave-7kzw"
server_key: "b9537741703b9d307da2eb5295f726fe"
server_key_iv: "a1ea5a37567bea422011f98d33176ecd"
server_key_version: "encryptionkey@0"
I20260806 16:06:37.719550 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:37.742938 734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } attempt: 1
I20260806 16:06:37.743827 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.743482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43056)
0806 16:06:37.743674 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.743737 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 16:06:37.744460 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:37.744793 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.744426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43068)
0806 16:06:37.744554 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.744695 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 16:06:37.744936 734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:33583: Network error: Client connection negotiation failed: client connection to 127.13.148.60:33583: connect: Connection refused (error 111)
I20260806 16:06:37.748013 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:37.748219 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:37.755488 847 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:33583
I20260806 16:06:37.755524 936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:33583 every 8 connection(s)
I20260806 16:06:37.756469 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
I20260806 16:06:37.759622 938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:37.761842 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 847
I20260806 16:06:37.762004 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786032227601881-13904-0/minicluster-data/master-2/wal/instance
I20260806 16:06:37.762326 13904 external_mini_cluster.cc:1428] Setting key 93795d6b5a11b71a5788c178bfdd0cd4
I20260806 16:06:37.764353 938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:06:37.772507 835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } attempt: 3
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:37.778417 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.766135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.13.148.62:46491)
0806 16:06:37.766314 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.766320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.766948 (+ 628us) server_negotiation.cc:407] Connection header received
0806 16:06:37.767162 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.767171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.767220 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.767330 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.768066 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.768944 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.770179 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.770592 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.770680 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.774036 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.774040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.774043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.774072 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.775898 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.776483 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.776487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.776488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.776572 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.777246 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.777250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.777252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.777479 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.777561 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.778083 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.778224 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:06:37.779417 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.763900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.13.148.60:33583)
0806 16:06:37.764358 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.764373 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.764411 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:06:37.765190 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.765211 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.765553 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.765934 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.766996 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.768030 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.769289 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.769975 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.770189 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.773040 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.773049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.773063 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.773092 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.775502 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.776484 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.776490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.776492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.776618 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.776939 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.776944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.776946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.777505 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.777729 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.778810 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.778949 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
I20260806 16:06:37.779688 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.766051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46491 (local address 127.0.0.1:59410)
0806 16:06:37.766805 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.766865 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.767043 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.767383 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.767389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.767754 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.767915 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.767925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.769096 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.769099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.770064 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.770073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.770764 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.770771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.770891 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.771436 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.771453 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.773873 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.776010 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.776019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.776032 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.776351 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.776965 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.776967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.776969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.777103 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.777663 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.777672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.777814 (+ 142us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.778005 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.778351 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":165,"threads_started":1}
I20260806 16:06:37.781016 938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:06:37.784878 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.773003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43072)
0806 16:06:37.773206 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.773221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.773308 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.773703 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.773706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.773906 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.774045 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.774054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.775827 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.775834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.776791 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.776801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.777294 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.777298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.777396 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.777933 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.777949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.779471 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.783135 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.783138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.783141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.783445 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.783753 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.783756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.783758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.783859 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.784321 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.784325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.784437 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.784558 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.784706 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260806 16:06:37.784915 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.773099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.13.148.60:33583)
0806 16:06:37.773430 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.773441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.773454 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:06:37.773495 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.773497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.773530 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.773645 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.774175 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.775719 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.776909 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.777171 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.777286 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.780631 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.780634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.780636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.780658 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.783017 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.783571 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.783575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.783577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.783670 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.783958 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.783962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.783964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.784215 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.784328 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.784634 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.784744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":91,"threads_started":1}
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:37.788269 835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c: Bootstrap starting.
I20260806 16:06:37.791467 835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:37.791882 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.781842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.13.148.61:41905)
0806 16:06:37.781982 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.781990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.782009 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:06:37.782097 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.782101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.782150 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.782251 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.782886 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.783550 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.784431 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.784798 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.784902 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.787941 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.787945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.787948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.787979 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.789912 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.790395 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.790401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.790405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.790532 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.790862 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.790865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.790867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.791102 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.791177 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.791598 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.791730 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:06:37.792375 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.781746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41905 (local address 127.0.0.1:55568)
0806 16:06:37.781927 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.781941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.782009 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.782382 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.782389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.782610 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.782723 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.782733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.783664 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.783667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.784329 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.784338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.785420 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.785424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.785522 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.785953 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.785969 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.787783 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.790040 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.790043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.790045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.790280 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.790646 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.790649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.790651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.790746 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.791256 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.791261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.791369 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.792061 (+ 692us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.792201 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 16:06:37.792440 835 log.cc:824] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:37.792941 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.782249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.13.148.62:46491)
0806 16:06:37.782374 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.782379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.782392 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:06:37.782433 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.782437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.782511 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.782604 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.783287 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.784070 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.785321 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.785676 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.785775 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.788952 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.788956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.788958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.788985 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.790719 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.791147 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.791152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.791154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.791248 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.791540 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.791544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.791546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.791773 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.792045 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.792419 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.792567 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 16:06:37.793275 938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:06:37.795073 835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c: No bootstrap required, opened a new log
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:06:37.799515 835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:37.799798 835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:37.799849 835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d732fb5382a4b68aabae6f5db9d365c, State: Initialized, Role: FOLLOWER
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:06:37.800397 835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:37.800967 946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:37.801154 946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:06:37.801910 835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:06:37.804509 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.793826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43084)
0806 16:06:37.793920 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.793936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.794012 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.794279 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.794282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.794452 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.794587 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.794596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.795507 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.795510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.796231 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.796241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.796886 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.796890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.796970 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.797440 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.797454 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.798914 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.802680 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.802687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.802691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.802971 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.803329 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.803333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.803336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.803446 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.804001 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.804006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.804110 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.804219 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.804358 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 16:06:37.804463 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.794231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.13.148.61:41905)
0806 16:06:37.794326 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.794330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.794526 (+ 196us) server_negotiation.cc:407] Connection header received
0806 16:06:37.794636 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.794642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.794707 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.794783 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.795484 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.795999 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.797086 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.797474 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.797601 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.800552 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.800556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.800559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.800583 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.802337 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.802940 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.802944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.802946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.803048 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.803426 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.803430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.803432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.803695 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.803882 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.804144 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.804295 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:06:37.805146 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.793979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.13.148.60:33583)
0806 16:06:37.794092 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.794096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.794108 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:06:37.794136 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.794139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.794180 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.794267 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.794697 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.795389 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.796356 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.796763 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.796872 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.799059 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.799064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.799067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.799096 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.802561 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.803112 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.803117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.803121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.803220 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.803552 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.803557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.803559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.803853 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.803957 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.804848 (+ 891us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.804985 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 16:06:37.808302 938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: Bootstrap starting.
I20260806 16:06:37.810678 938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:37.806839 743 master_runner.cc:186] Error getting master registration for 127.13.148.62:46491: OK, instance_id { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" instance_seqno: 1786032397270641 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:06:37.811051 743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:06:37.811688 938 log.cc:824] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:37.812094 734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } attempt: 2
W20260806 16:06:37.812650 959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:06:37.812776 959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:06:37.813738 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:06:37.814761 938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: No bootstrap required, opened a new log
Aug 06 16:06:37 dist-test-slave-7kzw krb5kdc[614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032397, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:06:37.819451 938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:37.819784 938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:37.819861 938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a41a2bb731c4984842b8e05a41c6d4a, State: Initialized, Role: FOLLOWER
I20260806 16:06:37.820423 938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:37.821982 938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:06:37.823060 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.812513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:33583 (local address 127.0.0.1:43086)
0806 16:06:37.812684 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:37.812700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:37.812797 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:37.813154 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:37.813157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:37.813368 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:37.813476 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.813483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.814518 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.814522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.815416 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:37.815427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.816088 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.816091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:37.816184 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.816734 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:37.816748 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:37.818342 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:37.821179 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.821182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.821184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.821443 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.821737 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:37.821741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:37.821744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:37.821844 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:37.822375 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:37.822378 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:37.822619 (+ 241us) client_negotiation.cc:769] Sending connection context
0806 16:06:37.822733 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 16:06:37.822887 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 16:06:37.823742 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.812560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.13.148.60:33583)
0806 16:06:37.812672 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.812678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.812909 (+ 231us) server_negotiation.cc:407] Connection header received
0806 16:06:37.812958 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.812961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.813009 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.813106 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.813614 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.814358 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.815554 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.815855 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.815969 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.818504 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.818507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.818510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.818533 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.821053 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.821550 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.821553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.821555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.821635 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.821947 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.821950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.821952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.822187 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.822292 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.823447 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.823571 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":1792}
I20260806 16:06:37.825376 972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:37.826658 972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:06:37.828948 734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: Bootstrap starting.
I20260806 16:06:37.829386 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.816775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.13.148.60:33583)
0806 16:06:37.816881 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.816887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.819720 (+ 2833us) server_negotiation.cc:407] Connection header received
0806 16:06:37.819875 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.819878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.819916 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.819961 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.820921 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.821687 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.822820 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.823192 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.823285 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.825170 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.825174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.825177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.825200 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.827499 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.828002 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.828005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.828007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.828089 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.828395 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.828399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.828400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.828597 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.828708 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.829062 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.829176 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":768}
I20260806 16:06:37.827116 847 master_runner.cc:186] Error getting master registration for 127.13.148.62:46491: OK, instance_id { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" instance_seqno: 1786032397270641 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:06:37.830822 847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 16:06:37.829203 986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:06:37.830945 986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:06:37.832669 734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:37.834331 734 log.cc:824] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:37.836102 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.816081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.13.148.62:46491)
0806 16:06:37.816184 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.816188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.818149 (+ 1961us) server_negotiation.cc:407] Connection header received
0806 16:06:37.818337 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.818340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.818386 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.818438 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.819510 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.820220 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.825086 (+ 4866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.825438 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.825515 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.831942 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.831946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.831949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.831976 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.833940 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.834658 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.834662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.834664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.834756 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.835146 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.835149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.835151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.835368 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.835484 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.835825 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.835949 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 16:06:37.837661 734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: No bootstrap required, opened a new log
I20260806 16:06:37.838546 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:37.817587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.13.148.61:41905)
0806 16:06:37.817755 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:37.817759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:37.824107 (+ 6348us) server_negotiation.cc:407] Connection header received
0806 16:06:37.825671 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:37.825676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:37.825730 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:37.825783 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:37.830425 (+ 4642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.831183 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.832235 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:37.832541 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:37.832641 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:37.834313 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:37.834317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:37.834320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:37.834353 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:37.836370 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.836941 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.836945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.836947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.837059 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:37.837426 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:37.837430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:37.837432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:37.837660 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:37.837804 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:37.838227 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 16:06:37.838356 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 16:06:37.841645 734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:37.841869 734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:37.841928 734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5331a6d261c74fbaa0b425aaac4fe5e3, State: Initialized, Role: FOLLOWER
I20260806 16:06:37.842430 734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:37.844269 734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 16:06:37.851359 1002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:06:37.851519 1002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:06:37.854353 632 master_runner.cc:263] No leader master found from master 5331a6d261c74fbaa0b425aaac4fe5e3
I20260806 16:06:37.854427 632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:06:37.854640 990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:37.854774 990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 16:06:37.999663 872 tablet.cc:2369] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:06:38.012077 770 tablet.cc:2369] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:06:38.016827 672 tablet.cc:2369] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:06:38.028431 946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:38.028672 946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:38.029672 946 leader_election.cc:288] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5331a6d261c74fbaa0b425aaac4fe5e3 (127.13.148.62:46491), 1a41a2bb731c4984842b8e05a41c6d4a (127.13.148.60:33583)
I20260806 16:06:38.030277 698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d732fb5382a4b68aabae6f5db9d365c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" is_pre_election: true
I20260806 16:06:38.030288 902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d732fb5382a4b68aabae6f5db9d365c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" is_pre_election: true
I20260806 16:06:38.030630 902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d732fb5382a4b68aabae6f5db9d365c in term 0.
I20260806 16:06:38.030630 698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d732fb5382a4b68aabae6f5db9d365c in term 0.
I20260806 16:06:38.031087 767 leader_election.cc:302] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d732fb5382a4b68aabae6f5db9d365c, 5331a6d261c74fbaa0b425aaac4fe5e3; no voters:
I20260806 16:06:38.031390 946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:06:38.031477 946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:06:38.031522 946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:38.032585 946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:38.032938 946 leader_election.cc:288] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [CANDIDATE]: Term 1 election: Requested vote from peers 5331a6d261c74fbaa0b425aaac4fe5e3 (127.13.148.62:46491), 1a41a2bb731c4984842b8e05a41c6d4a (127.13.148.60:33583)
I20260806 16:06:38.033317 902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d732fb5382a4b68aabae6f5db9d365c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a41a2bb731c4984842b8e05a41c6d4a"
I20260806 16:06:38.033317 698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d732fb5382a4b68aabae6f5db9d365c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3"
I20260806 16:06:38.033444 902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:38.033455 698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:38.034761 698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d732fb5382a4b68aabae6f5db9d365c in term 1.
I20260806 16:06:38.034755 902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d732fb5382a4b68aabae6f5db9d365c in term 1.
I20260806 16:06:38.035076 767 leader_election.cc:302] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a41a2bb731c4984842b8e05a41c6d4a, 2d732fb5382a4b68aabae6f5db9d365c; no voters:
I20260806 16:06:38.035363 946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:06:38.035609 946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [term 1 LEADER]: Becoming Leader. State: Replica: 2d732fb5382a4b68aabae6f5db9d365c, State: Running, Role: LEADER
I20260806 16:06:38.035952 946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } }
I20260806 16:06:38.038041 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d732fb5382a4b68aabae6f5db9d365c. Latest consensus state: current_term: 1 leader_uuid: "2d732fb5382a4b68aabae6f5db9d365c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:38.038164 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:38.038558 1006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:06:38.039525 1006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:06:38.045071 698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [term 1 FOLLOWER]: Refusing update from remote peer 2d732fb5382a4b68aabae6f5db9d365c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:06:38.045159 902 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [term 1 FOLLOWER]: Refusing update from remote peer 2d732fb5382a4b68aabae6f5db9d365c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:06:38.045789 1004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:06:38.046018 946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:06:38.049240 972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d732fb5382a4b68aabae6f5db9d365c. Latest consensus state: current_term: 1 leader_uuid: "2d732fb5382a4b68aabae6f5db9d365c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:38.049373 972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:06:38.049765 990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d732fb5382a4b68aabae6f5db9d365c. Latest consensus state: current_term: 1 leader_uuid: "2d732fb5382a4b68aabae6f5db9d365c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:38.049892 990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:06:38.052659 1016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d732fb5382a4b68aabae6f5db9d365c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:38.052691 946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d732fb5382a4b68aabae6f5db9d365c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:38.052830 946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:38.052784 1016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:38.053242 990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d732fb5382a4b68aabae6f5db9d365c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:38.053359 990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:06:38.053227 1011 mvcc.cc:205] Tried to move back new op lower bound from 7315588702383554560 to 7315588702358306816. Current Snapshot: MvccSnapshot[applied={T|T < 7315588702383554560}]
I20260806 16:06:38.053457 1008 mvcc.cc:205] Tried to move back new op lower bound from 7315588702383554560 to 7315588702358306816. Current Snapshot: MvccSnapshot[applied={T|T < 7315588702383554560}]
I20260806 16:06:38.054307 972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d732fb5382a4b68aabae6f5db9d365c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5331a6d261c74fbaa0b425aaac4fe5e3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46491 } } peers { permanent_uuid: "2d732fb5382a4b68aabae6f5db9d365c" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41905 } } peers { permanent_uuid: "1a41a2bb731c4984842b8e05a41c6d4a" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 33583 } } }
I20260806 16:06:38.054451 972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:06:38.055105 1006 catalog_manager.cc:1347] Generated new cluster ID: 10233680e3014456b3d075d50be286c3
I20260806 16:06:38.055189 1006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:06:38.079053 1006 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:06:38.080679 1006 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:06:38.091508 1006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d732fb5382a4b68aabae6f5db9d365c: Generated new TSK 0
I20260806 16:06:38.092341 1006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:06:38.832343 986 catalog_manager.cc:1259] Loaded cluster ID: 10233680e3014456b3d075d50be286c3
I20260806 16:06:38.832485 986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: loading cluster ID for follower catalog manager: success
I20260806 16:06:38.834383 986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: acquiring CA information for follower catalog manager: success
I20260806 16:06:38.835559 986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a41a2bb731c4984842b8e05a41c6d4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:06:38.853046 1002 catalog_manager.cc:1259] Loaded cluster ID: 10233680e3014456b3d075d50be286c3
I20260806 16:06:38.853137 1002 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: loading cluster ID for follower catalog manager: success
I20260806 16:06:38.855023 1002 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: acquiring CA information for follower catalog manager: success
I20260806 16:06:38.856248 1002 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5331a6d261c74fbaa0b425aaac4fe5e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:06:39.158401 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 632
I20260806 16:06:39.173568 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 743
I20260806 16:06:39.184628 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 847
2026-08-06T16:06:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3478 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:06:39 dist-test-slave-7kzw krb5kdc[1030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:06:39 dist-test-slave-7kzw krb5kdc[1030](info): set up 2 sockets
Aug 06 16:06:39 dist-test-slave-7kzw krb5kdc[1030](info): commencing operation
krb5kdc: starting...
W20260806 16:06:40.408852 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:06:40 dist-test-slave-7kzw krb5kdc[1030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:06:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:06:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:06:40.550053 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42097
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:42097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:40.665333 1046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:40.665616 1046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:40.665666 1046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:40.669725 1046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:40.669791 1046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:40.669811 1046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:40.669828 1046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:40.669843 1046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:40.675148 1046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:42097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:40.676179 1046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:40.677245 1046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:40.685833 1059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:40.685906 1056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:40.685909 1055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:40.686062 1046 server_base.cc:1034] running on GCE node
I20260806 16:06:40.686906 1046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:40.688028 1046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:40.689189 1046 hybrid_clock.cc:648] HybridClock initialized: now 1786032400689176 us; error 26 us; skew 500 ppm
Aug 06 16:06:40 dist-test-slave-7kzw krb5kdc[1030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032400, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:40.692130 1046 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:06:40.693068 1046 webserver.cc:460] Webserver started at http://127.13.148.62:38313/ using document root <none> and password file <none>
I20260806 16:06:40.693637 1046 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:40.693701 1046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:40.693948 1046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:06:40.696066 1046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "0800fb1f553044b0b5d3dfdf1affba2a"
format_stamp: "Formatted at 2026-08-06 16:06:40 on dist-test-slave-7kzw"
server_key: "4f296d5837143cc9305a439ab3d98f7e"
server_key_iv: "ae88ffc0eae9736499655adb45b0d428"
server_key_version: "encryptionkey@0"
I20260806 16:06:40.696681 1046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "0800fb1f553044b0b5d3dfdf1affba2a"
format_stamp: "Formatted at 2026-08-06 16:06:40 on dist-test-slave-7kzw"
server_key: "4f296d5837143cc9305a439ab3d98f7e"
server_key_iv: "ae88ffc0eae9736499655adb45b0d428"
server_key_version: "encryptionkey@0"
I20260806 16:06:40.700862 1046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:06:40.703446 1068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:40.704777 1046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:06:40.704936 1046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "0800fb1f553044b0b5d3dfdf1affba2a"
format_stamp: "Formatted at 2026-08-06 16:06:40 on dist-test-slave-7kzw"
server_key: "4f296d5837143cc9305a439ab3d98f7e"
server_key_iv: "ae88ffc0eae9736499655adb45b0d428"
server_key_version: "encryptionkey@0"
I20260806 16:06:40.705060 1046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:40.723601 1046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:40.726871 1046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:40.727119 1046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:40.734691 1046 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:42097
I20260806 16:06:40.734716 1135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:42097 every 8 connection(s)
I20260806 16:06:40.735708 1046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:06:40.737457 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 1046
I20260806 16:06:40.737586 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:06:40.737888 13904 external_mini_cluster.cc:1428] Setting key 650347721d3e16e31a7069b099f3a554
I20260806 16:06:40.738778 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:40.745608 1137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a: Bootstrap starting.
Aug 06 16:06:40 dist-test-slave-7kzw krb5kdc[1030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:40.748193 1137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:40.749109 1137 log.cc:824] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:40.751662 1137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a: No bootstrap required, opened a new log
I20260806 16:06:40.754084 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:40.739340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.13.148.62:42097)
0806 16:06:40.739776 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:40.739795 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:40.739840 (+ 45us) server_negotiation.cc:407] Connection header received
0806 16:06:40.740712 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:40.740735 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:40.741147 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:40.741533 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:40.742278 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:40.743824 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:40.744818 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:40.745501 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:40.745664 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:40.748601 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:40.748608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:40.748626 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:40.748660 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:40.750928 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:40.751383 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:40.751389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:40.751395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:40.751546 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:40.751836 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:40.751840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:40.751841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:40.752376 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:40.752572 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:40.752777 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 16:06:40.753073 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":123,"threads_started":1}
I20260806 16:06:40.756078 1137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800fb1f553044b0b5d3dfdf1affba2a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42097 } }
I20260806 16:06:40.756351 1137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:40.756421 1137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0800fb1f553044b0b5d3dfdf1affba2a, State: Initialized, Role: FOLLOWER
I20260806 16:06:40.756889 1137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800fb1f553044b0b5d3dfdf1affba2a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42097 } }
I20260806 16:06:40.757052 1137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:06:40.757128 1137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:06:40.757241 1137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:40.757412 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:06:40.758234 1137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800fb1f553044b0b5d3dfdf1affba2a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42097 } }
I20260806 16:06:40.758641 1137 leader_election.cc:302] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0800fb1f553044b0b5d3dfdf1affba2a; no voters:
I20260806 16:06:40.758998 1137 leader_election.cc:288] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:06:40.759266 1145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:06:40.759516 1145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [term 1 LEADER]: Becoming Leader. State: Replica: 0800fb1f553044b0b5d3dfdf1affba2a, State: Running, Role: LEADER
I20260806 16:06:40.759862 1145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800fb1f553044b0b5d3dfdf1affba2a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42097 } }
I20260806 16:06:40.760291 1137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:06:40.761518 1146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0800fb1f553044b0b5d3dfdf1affba2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800fb1f553044b0b5d3dfdf1affba2a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42097 } } }
I20260806 16:06:40.761646 1146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:40.762032 1156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:06:40.762032 1149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0800fb1f553044b0b5d3dfdf1affba2a. Latest consensus state: current_term: 1 leader_uuid: "0800fb1f553044b0b5d3dfdf1affba2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0800fb1f553044b0b5d3dfdf1affba2a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42097 } } }
I20260806 16:06:40.762691 1149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:40.765347 1156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:06:40.771039 1156 catalog_manager.cc:1347] Generated new cluster ID: 1f8752f9d3304b5dad98633641768687
I20260806 16:06:40.771114 1156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:06:40.779453 1156 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:06:40.780409 1156 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:06:40.787796 1156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0800fb1f553044b0b5d3dfdf1affba2a: Generated new TSK 0
I20260806 16:06:40.788515 1156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:06:40.814657 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 1046
2026-08-06T16:06:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1628 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:06:40 dist-test-slave-7kzw krb5kdc[1172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:06:40 dist-test-slave-7kzw krb5kdc[1172](info): set up 2 sockets
Aug 06 16:06:40 dist-test-slave-7kzw krb5kdc[1172](info): commencing operation
krb5kdc: starting...
W20260806 16:06:42.057300 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:06:42 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:06:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:06:42.193547 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39593
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:42.307788 1194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:42.308159 1194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:42.308247 1194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:42.312427 1194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:42.312536 1194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:42.312575 1194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:42.312623 1194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:42.312659 1194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:42.318080 1194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:42.319264 1194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:42.320376 1194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:42.329497 1203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:42.329497 1205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:42.329612 1202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:42.329924 1194 server_base.cc:1034] running on GCE node
I20260806 16:06:42.330592 1194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:42.331799 1194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:42.333014 1194 hybrid_clock.cc:648] HybridClock initialized: now 1786032402332999 us; error 27 us; skew 500 ppm
Aug 06 16:06:42 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032402, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:42.335927 1194 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:06:42.336869 1194 webserver.cc:460] Webserver started at http://127.13.148.62:35227/ using document root <none> and password file <none>
I20260806 16:06:42.337415 1194 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:42.337477 1194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:42.337661 1194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:06:42.339947 1194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "73a4743249f842fd980b83bee23698dd"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "ebc543aca44f27aa94793dcc923088cb"
server_key_iv: "345a31a860cf95ed79f1a9a845ddb107"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.340515 1194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "73a4743249f842fd980b83bee23698dd"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "ebc543aca44f27aa94793dcc923088cb"
server_key_iv: "345a31a860cf95ed79f1a9a845ddb107"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.344740 1194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:06:42.348286 1218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:42.349687 1194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:06:42.349833 1194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "73a4743249f842fd980b83bee23698dd"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "ebc543aca44f27aa94793dcc923088cb"
server_key_iv: "345a31a860cf95ed79f1a9a845ddb107"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.349933 1194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:42.362056 1194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:42.365244 1194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:42.365413 1194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:42.373453 1194 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39593
I20260806 16:06:42.373864 1290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39593 every 8 connection(s)
I20260806 16:06:42.374641 1194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:06:42.377713 1291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:42.381318 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 1194
I20260806 16:06:42.381475 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:06:42.381803 13904 external_mini_cluster.cc:1428] Setting key c1ef69868e650d80be5317e6b81aa2e1
I20260806 16:06:42.384490 1291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Bootstrap starting.
I20260806 16:06:42.387360 1291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:42.388156 1291 log.cc:824] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:42.390251 1291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: No bootstrap required, opened a new log
Aug 06 16:06:42 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:42.394322 1291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:42.394755 1291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:42.394876 1291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a4743249f842fd980b83bee23698dd, State: Initialized, Role: FOLLOWER
I20260806 16:06:42.395350 1291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:42.395515 1291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:06:42.395596 1291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:06:42.395704 1291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:42.396929 1291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:42.397230 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:42.383357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.13.148.62:39593)
0806 16:06:42.383787 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:42.383810 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:42.383853 (+ 43us) server_negotiation.cc:407] Connection header received
0806 16:06:42.384489 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:42.384505 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:42.384891 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:42.385175 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:42.386282 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.387461 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.388392 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.388877 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.389071 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:42.392053 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:42.392058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:42.392072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:42.392100 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:42.393863 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:42.394360 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:42.394366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:42.394371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:42.394548 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:42.394953 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:42.394956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:42.394958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:42.395462 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:42.395703 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:42.396045 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 16:06:42.396394 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":146,"threads_started":1}
I20260806 16:06:42.397307 1291 leader_election.cc:302] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a4743249f842fd980b83bee23698dd; no voters:
I20260806 16:06:42.397715 1291 leader_election.cc:288] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:06:42.397792 1307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:06:42.398000 1307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 1 LEADER]: Becoming Leader. State: Replica: 73a4743249f842fd980b83bee23698dd, State: Running, Role: LEADER
I20260806 16:06:42.398361 1307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:42.399101 1291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:06:42.399652 1307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73a4743249f842fd980b83bee23698dd. Latest consensus state: current_term: 1 leader_uuid: "73a4743249f842fd980b83bee23698dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } } }
I20260806 16:06:42.399767 1307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:42.400087 1308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73a4743249f842fd980b83bee23698dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } } }
I20260806 16:06:42.400218 1308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:42.400494 1313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:06:42.404021 1313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:06:42.410197 1313 catalog_manager.cc:1347] Generated new cluster ID: 6ac3d370b8a64537b18ea6edeb4e26b3
I20260806 16:06:42.410274 1313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:06:42.423625 1313 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:06:42.425016 1313 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:06:42.434115 1313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Generated new TSK 0
I20260806 16:06:42.435009 1313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:06:42.471066 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39593
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:42.594009 1338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:42.594316 1338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:42.594395 1338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:42.598336 1338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:42.598428 1338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:42.598549 1338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:06:42.604158 1338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:42.605225 1338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:42.606236 1338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:42.616009 1346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:42.615974 1344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:42.616274 1338 server_base.cc:1034] running on GCE node
W20260806 16:06:42.616019 1348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:42.616684 1338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:42.617480 1338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:42.618647 1338 hybrid_clock.cc:648] HybridClock initialized: now 1786032402618627 us; error 30 us; skew 500 ppm
Aug 06 16:06:42 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032402, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:42.621503 1338 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:06:42.622305 1338 webserver.cc:460] Webserver started at http://127.13.148.1:36327/ using document root <none> and password file <none>
I20260806 16:06:42.622893 1338 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:42.622959 1338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:42.623126 1338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:06:42.625123 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "1679b864b5d44183b9233c55f5913ef4"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "748cf9220b2ae448c02ffe152d3ea109"
server_key_iv: "c6379de997add55f6c7ba02e43f26398"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.625658 1338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "1679b864b5d44183b9233c55f5913ef4"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "748cf9220b2ae448c02ffe152d3ea109"
server_key_iv: "c6379de997add55f6c7ba02e43f26398"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.629580 1338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 16:06:42.632081 1358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:42.633582 1338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:06:42.633723 1338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "1679b864b5d44183b9233c55f5913ef4"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "748cf9220b2ae448c02ffe152d3ea109"
server_key_iv: "c6379de997add55f6c7ba02e43f26398"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.633837 1338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:42.653280 1338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:42.657068 1338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:42.657274 1338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:42.657976 1338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:42.658932 1338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:42.658983 1338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:42.659026 1338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:42.659057 1338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:42.668146 1338 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:43851
I20260806 16:06:42.668177 1518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:43851 every 8 connection(s)
I20260806 16:06:42.669057 1338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:06:42.669554 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 1338
I20260806 16:06:42.669695 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:06:42.670049 13904 external_mini_cluster.cc:1428] Setting key 5ea6d3082100ce62ea05d43f07148b23
Aug 06 16:06:42 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032402, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:06:42.685322 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:42.671041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:59515 (local address 127.13.148.62:39593)
0806 16:06:42.671203 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:42.671208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:42.672102 (+ 894us) server_negotiation.cc:407] Connection header received
0806 16:06:42.673498 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:42.673504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:42.673569 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:42.673668 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:42.675498 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.676498 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.677828 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.678266 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.678406 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:42.681179 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:42.681185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:42.681188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:42.681221 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:42.682885 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:42.683515 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:42.683518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:42.683519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:42.683598 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:42.683978 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:42.683982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:42.683984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:42.684219 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:42.684334 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:42.684908 (+ 574us) server_negotiation.cc:299] Negotiation successful
0806 16:06:42.685059 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:06:42.686034 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:42.671459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:59515)
0806 16:06:42.671936 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:42.671988 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:42.673291 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:42.673946 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:42.673959 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:42.674530 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:42.675308 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:42.675329 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.676659 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.676662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:42.677681 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:42.677692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.678423 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.678426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:42.678638 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:42.679413 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:42.679429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:42.680991 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:42.683047 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:42.683057 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:42.683075 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:42.683422 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:42.683710 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:42.683713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:42.683714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:42.683860 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:42.684352 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:42.684358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:42.684596 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 16:06:42.684770 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 16:06:42.685130 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":139,"threads_started":1}
I20260806 16:06:42.687595 1519 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39593
I20260806 16:06:42.687856 1519 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:42.688364 1519 heartbeater.cc:511] Master 127.13.148.62:39593 requested a full tablet report, sending...
I20260806 16:06:42.689878 1238 ts_manager.cc:194] Registered new tserver with Master: 1679b864b5d44183b9233c55f5913ef4 (127.13.148.1:43851)
I20260806 16:06:42.691134 1238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:59515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:06:42.702313 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39593
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:42.817625 1527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:42.817906 1527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:42.817952 1527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:42.822050 1527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:42.822122 1527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:42.822187 1527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:06:42.827479 1527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:42.828601 1527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:42.829633 1527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:06:42.839939 1527 server_base.cc:1034] running on GCE node
W20260806 16:06:42.839903 1540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:42.839896 1535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:42.839892 1533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:42.840648 1527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:42.841394 1527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:42.842554 1527 hybrid_clock.cc:648] HybridClock initialized: now 1786032402842527 us; error 45 us; skew 500 ppm
Aug 06 16:06:42 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032402, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:42.849359 1527 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:06:42.850390 1527 webserver.cc:460] Webserver started at http://127.13.148.2:39265/ using document root <none> and password file <none>
I20260806 16:06:42.851054 1527 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:42.851130 1527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:42.851323 1527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:06:42.853446 1527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "c106164208bf46c5b673c9fdca76e2ae"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "bfaaf301fd9dce00c8408f39eed03eeb"
server_key_iv: "413a492584263b0e76ba04c9e78aaf3e"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.853978 1527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "c106164208bf46c5b673c9fdca76e2ae"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "bfaaf301fd9dce00c8408f39eed03eeb"
server_key_iv: "413a492584263b0e76ba04c9e78aaf3e"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.857868 1527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:06:42.860522 1551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:42.861896 1527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:06:42.862051 1527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "c106164208bf46c5b673c9fdca76e2ae"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "bfaaf301fd9dce00c8408f39eed03eeb"
server_key_iv: "413a492584263b0e76ba04c9e78aaf3e"
server_key_version: "encryptionkey@0"
I20260806 16:06:42.862169 1527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:42.874800 1527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:42.878331 1527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:42.878634 1527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:42.879398 1527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:42.880398 1527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:42.880479 1527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:42.880550 1527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:42.880604 1527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:42.890167 1527 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:43725
I20260806 16:06:42.890185 1704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:43725 every 8 connection(s)
I20260806 16:06:42.891228 1527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:06:42.900970 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 1527
I20260806 16:06:42.901088 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:06:42.901448 13904 external_mini_cluster.cc:1428] Setting key 9580d92bd7b7e42ae26aa513c4fa14c1
Aug 06 16:06:42 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032402, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:42.907711 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:42.893167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:55385 (local address 127.13.148.62:39593)
0806 16:06:42.893328 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:42.893332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:42.894215 (+ 883us) server_negotiation.cc:407] Connection header received
0806 16:06:42.895745 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:42.895748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:42.895816 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:42.896022 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:42.897506 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.898110 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.899479 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.899743 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.899880 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:42.903086 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:42.903090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:42.903093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:42.903122 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:42.904899 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:42.905715 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:42.905721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:42.905723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:42.905832 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:42.906220 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:42.906225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:42.906228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:42.906528 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:42.906637 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:42.907259 (+ 622us) server_negotiation.cc:299] Negotiation successful
0806 16:06:42.907436 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 16:06:42.908497 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:42.893570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:55385)
0806 16:06:42.894050 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:42.894094 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:42.895483 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:42.896320 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:42.896329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:42.896784 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:42.897323 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:42.897340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.898263 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.898267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:42.899265 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:42.899276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:42.899931 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:42.899936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:42.900150 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:42.900973 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:42.900991 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:42.902923 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:42.905035 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:42.905045 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:42.905068 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:42.905529 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:42.905943 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:42.905946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:42.905947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:42.906106 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:42.906687 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:42.906693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:42.906962 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 16:06:42.907151 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 16:06:42.907467 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":131,"threads_started":1}
I20260806 16:06:42.909943 1706 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39593
I20260806 16:06:42.910223 1706 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:42.910782 1706 heartbeater.cc:511] Master 127.13.148.62:39593 requested a full tablet report, sending...
I20260806 16:06:42.912089 1238 ts_manager.cc:194] Registered new tserver with Master: c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725)
I20260806 16:06:42.912998 1238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:55385
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:06:42.935928 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39593
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:43.066641 1713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:43.066897 1713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:43.066965 1713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:43.071010 1713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:43.071072 1713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:43.071132 1713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:06:43.076251 1713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:43.077296 1713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:43.078315 1713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:43.088304 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:43.088315 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:43.088315 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:43.088644 1713 server_base.cc:1034] running on GCE node
I20260806 16:06:43.089198 1713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:43.089828 1713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:43.091008 1713 hybrid_clock.cc:648] HybridClock initialized: now 1786032403090976 us; error 48 us; skew 500 ppm
Aug 06 16:06:43 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032403, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:43.093982 1713 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:06:43.094937 1713 webserver.cc:460] Webserver started at http://127.13.148.3:42233/ using document root <none> and password file <none>
I20260806 16:06:43.095506 1713 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:43.095574 1713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:43.095808 1713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:06:43.098266 1713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "5315bfa3b8414931999644babe6e6a48"
format_stamp: "Formatted at 2026-08-06 16:06:43 on dist-test-slave-7kzw"
server_key: "71c39efc9925b790fa90540fba0a1c2d"
server_key_iv: "e42deb42bde117f8fee6c695c3eb9391"
server_key_version: "encryptionkey@0"
I20260806 16:06:43.099094 1713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "5315bfa3b8414931999644babe6e6a48"
format_stamp: "Formatted at 2026-08-06 16:06:43 on dist-test-slave-7kzw"
server_key: "71c39efc9925b790fa90540fba0a1c2d"
server_key_iv: "e42deb42bde117f8fee6c695c3eb9391"
server_key_version: "encryptionkey@0"
I20260806 16:06:43.103345 1713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:06:43.105813 1733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:43.107244 1713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:06:43.107414 1713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "5315bfa3b8414931999644babe6e6a48"
format_stamp: "Formatted at 2026-08-06 16:06:43 on dist-test-slave-7kzw"
server_key: "71c39efc9925b790fa90540fba0a1c2d"
server_key_iv: "e42deb42bde117f8fee6c695c3eb9391"
server_key_version: "encryptionkey@0"
I20260806 16:06:43.107535 1713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:43.132309 1713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:43.136313 1713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:43.136565 1713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:43.137283 1713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:43.138347 1713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:43.138406 1713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:43.138520 1713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:43.138571 1713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 16:06:43.149205 1713 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36051
I20260806 16:06:43.149227 1888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36051 every 8 connection(s)
I20260806 16:06:43.150244 1713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:06:43.154238 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 1713
I20260806 16:06:43.154415 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:06:43.154778 13904 external_mini_cluster.cc:1428] Setting key 5be9b4d6b30f9dbad0ba7e2590203607
Aug 06 16:06:43 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032403, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:43.165711 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.152267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:48943 (local address 127.13.148.62:39593)
0806 16:06:43.152444 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.152448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.153386 (+ 938us) server_negotiation.cc:407] Connection header received
0806 16:06:43.155179 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.155183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.155236 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.155340 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:43.156784 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.157456 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.158738 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.158986 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.159117 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.161892 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:43.161896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:43.161899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:43.161936 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:43.163476 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:43.164050 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:43.164054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:43.164055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:43.164147 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:43.164505 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:43.164510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:43.164511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:43.164718 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:43.164832 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.165396 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.165528 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 16:06:43.166692 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.152607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:48943)
0806 16:06:43.153191 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.153264 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:43.154927 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:43.155509 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:43.155517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:43.156002 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:43.156616 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.156631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.157611 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.157615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.158591 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.158602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.159156 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.159160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.159349 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.160051 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:43.160070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:43.161702 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:43.163615 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:43.163621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:43.163638 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:43.163931 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:43.164244 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:43.164248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:43.164250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:43.164403 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:43.164830 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:43.164838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:43.165103 (+ 265us) client_negotiation.cc:769] Sending connection context
0806 16:06:43.165354 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 16:06:43.165707 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":116,"threads_started":1}
I20260806 16:06:43.168141 1889 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39593
I20260806 16:06:43.168442 1889 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:43.169179 1889 heartbeater.cc:511] Master 127.13.148.62:39593 requested a full tablet report, sending...
I20260806 16:06:43.170336 1238 ts_manager.cc:194] Registered new tserver with Master: 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
I20260806 16:06:43.170957 1238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:48943
I20260806 16:06:43.181574 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:06:43.191401 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.183718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.13.148.62:39593)
0806 16:06:43.183863 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.183867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.184021 (+ 154us) server_negotiation.cc:407] Connection header received
0806 16:06:43.184141 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.184143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.184184 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.184244 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:43.185014 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.185586 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.186425 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.186826 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.186919 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.188307 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:43.188311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:43.188313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:43.188338 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:43.189774 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:43.190171 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:43.190173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:43.190175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:43.190271 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:43.190591 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:43.190595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:43.190596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:43.190836 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:43.190954 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.191177 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.191280 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:06:43.194300 1238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:06:43.197459 1238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:06:43.213361 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.207302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36051 (local address 127.0.0.1:37398)
0806 16:06:43.208300 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.208316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:43.208423 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:43.208962 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:43.208965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:43.208983 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:43.209106 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.209115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.210095 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.210101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.211502 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.211511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.212876 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.212882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.212984 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.213008 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:06:43.213089 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 16:06:43.213169 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":16,"spinlock_wait_cycles":3840,"thread_start_us":121,"threads_started":1}
I20260806 16:06:43.213362 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.207765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.13.148.2:43725)
0806 16:06:43.208085 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.208099 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.208122 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:06:43.208201 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.208205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.208398 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.208563 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:43.208860 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.210044 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.211699 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.212496 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.212648 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.212774 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.213085 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.213184 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":106,"threads_started":1}
I20260806 16:06:43.213398 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.207499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.13.148.3:36051)
0806 16:06:43.208032 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.208046 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.208524 (+ 478us) server_negotiation.cc:407] Connection header received
0806 16:06:43.208600 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.208605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.208809 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.208987 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:43.209241 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.209958 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.211623 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.212516 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.212623 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.212774 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.213125 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.213226 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":203,"threads_started":1}
I20260806 16:06:43.213524 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.206614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43851 (local address 127.0.0.1:38092)
0806 16:06:43.207550 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.207561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:43.207652 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:43.208993 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:43.208997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:43.209018 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:43.209113 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.209119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.210668 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.210672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.211861 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.211870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.212747 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.212751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.213226 (+ 475us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.213257 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 16:06:43.213328 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 16:06:43.213390 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":394,"thread_start_us":107,"threads_started":1}
I20260806 16:06:43.213994 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.206543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:43725 (local address 127.0.0.1:54878)
0806 16:06:43.206911 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.206967 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:43.207103 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:43.208556 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:43.208563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:43.208585 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:43.208735 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.208741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.210194 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.210197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.211586 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.211597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.212678 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.212684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.212802 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.212868 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 16:06:43.212982 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 16:06:43.213113 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":74,"thread_start_us":104,"threads_started":1}
I20260806 16:06:43.214442 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.208234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.13.148.1:43851)
0806 16:06:43.208535 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.208541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.208563 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:06:43.208637 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.208641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.208838 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.209275 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:43.209412 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.210542 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.211981 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.212588 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.212856 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.213005 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.214188 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.214298 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
I20260806 16:06:43.216809 1618 tablet_service.cc:1467] Processing CreateTablet for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 (DEFAULT_TABLE table=test-table [id=e376bb7ec0954261ba031f0f3dbff6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:43.216921 1430 tablet_service.cc:1467] Processing CreateTablet for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 (DEFAULT_TABLE table=test-table [id=e376bb7ec0954261ba031f0f3dbff6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:43.216924 1793 tablet_service.cc:1467] Processing CreateTablet for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 (DEFAULT_TABLE table=test-table [id=e376bb7ec0954261ba031f0f3dbff6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:43.217980 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:43.217975 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:43.217976 1618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:43.224272 1911 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: Bootstrap starting.
I20260806 16:06:43.226866 1912 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Bootstrap starting.
I20260806 16:06:43.227205 1911 tablet_bootstrap.cc:654] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:43.228235 1911 log.cc:824] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:43.229158 1912 tablet_bootstrap.cc:654] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:43.229203 1913 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: Bootstrap starting.
I20260806 16:06:43.229981 1912 log.cc:824] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:43.230309 1911 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: No bootstrap required, opened a new log
I20260806 16:06:43.230564 1911 ts_tablet_manager.cc:1397] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:06:43.231815 1913 tablet_bootstrap.cc:654] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:43.232194 1912 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: No bootstrap required, opened a new log
I20260806 16:06:43.232385 1912 ts_tablet_manager.cc:1397] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:06:43.232667 1913 log.cc:824] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:43.233935 1911 raft_consensus.cc:348] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.234190 1911 raft_consensus.cc:374] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:43.234246 1911 raft_consensus.cc:729] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c106164208bf46c5b673c9fdca76e2ae, State: Initialized, Role: FOLLOWER
I20260806 16:06:43.234707 1911 consensus_queue.cc:260] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.234880 1913 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: No bootstrap required, opened a new log
I20260806 16:06:43.235066 1913 ts_tablet_manager.cc:1397] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:06:43.235762 1911 ts_tablet_manager.cc:1428] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:06:43.236325 1706 heartbeater.cc:503] Master 127.13.148.62:39593 was elected leader, sending a full tablet report...
I20260806 16:06:43.238129 1912 raft_consensus.cc:348] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.238404 1912 raft_consensus.cc:374] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:43.238488 1912 raft_consensus.cc:729] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679b864b5d44183b9233c55f5913ef4, State: Initialized, Role: FOLLOWER
I20260806 16:06:43.238950 1912 consensus_queue.cc:260] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.238874 1913 raft_consensus.cc:348] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.239109 1913 raft_consensus.cc:374] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:43.239187 1913 raft_consensus.cc:729] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5315bfa3b8414931999644babe6e6a48, State: Initialized, Role: FOLLOWER
I20260806 16:06:43.239657 1913 consensus_queue.cc:260] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.240013 1912 ts_tablet_manager.cc:1428] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Time spent starting tablet: real 0.008s user 0.000s sys 0.007s
I20260806 16:06:43.240345 1889 heartbeater.cc:503] Master 127.13.148.62:39593 was elected leader, sending a full tablet report...
I20260806 16:06:43.240345 1519 heartbeater.cc:503] Master 127.13.148.62:39593 was elected leader, sending a full tablet report...
I20260806 16:06:43.240589 1913 ts_tablet_manager.cc:1428] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:06:43.282665 1919 raft_consensus.cc:482] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:43.282948 1919 raft_consensus.cc:504] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.284482 1919 leader_election.cc:288] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1679b864b5d44183b9233c55f5913ef4 (127.13.148.1:43851), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
I20260806 16:06:43.289016 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.284723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43851 (local address 127.13.148.2:52121)
0806 16:06:43.284885 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.284905 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:43.285013 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:43.285481 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:43.285484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:43.285504 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:43.285679 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.285687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.286717 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.286721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.287659 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.287669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.288542 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.288546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.288669 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.288701 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 16:06:43.288779 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 16:06:43.288859 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":32}
I20260806 16:06:43.289067 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.284912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:52121 (local address 127.13.148.1:43851)
0806 16:06:43.285046 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.285054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.285085 (+ 31us) server_negotiation.cc:407] Connection header received
0806 16:06:43.285154 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.285195 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.285306 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.285451 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:43.285827 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.286586 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.287805 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.288398 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.288504 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.288554 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.288819 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.288905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:06:43.289238 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.285169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:43437 (local address 127.13.148.3:36051)
0806 16:06:43.285386 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.285395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.285558 (+ 163us) server_negotiation.cc:407] Connection header received
0806 16:06:43.285697 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.285700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.285757 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.285834 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:43.286164 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.286783 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.288114 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.288581 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.288750 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.288809 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.289013 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.289108 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 16:06:43.289638 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.285026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36051 (local address 127.13.148.2:43437)
0806 16:06:43.285458 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.285476 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:43.285600 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:43.285869 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:43.285872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:43.285893 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:43.286016 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.286024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.286910 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.286913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.287957 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.287969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.288730 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.288733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.288871 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.288895 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:06:43.288985 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:06:43.289056 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":3,"thread_start_us":153,"threads_started":1}
I20260806 16:06:43.289872 1459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca12ebcbf1fa4efeaa4ffa09fbb9cf44" candidate_uuid: "c106164208bf46c5b673c9fdca76e2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679b864b5d44183b9233c55f5913ef4" is_pre_election: true
I20260806 16:06:43.290222 1459 raft_consensus.cc:2454] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c106164208bf46c5b673c9fdca76e2ae in term 0.
I20260806 16:06:43.290309 1827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca12ebcbf1fa4efeaa4ffa09fbb9cf44" candidate_uuid: "c106164208bf46c5b673c9fdca76e2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5315bfa3b8414931999644babe6e6a48" is_pre_election: true
I20260806 16:06:43.290742 1827 raft_consensus.cc:2454] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c106164208bf46c5b673c9fdca76e2ae in term 0.
I20260806 16:06:43.290859 1552 leader_election.cc:302] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4, c106164208bf46c5b673c9fdca76e2ae; no voters:
I20260806 16:06:43.291186 1919 raft_consensus.cc:2781] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:06:43.291290 1919 raft_consensus.cc:482] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:06:43.291342 1919 raft_consensus.cc:3037] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:43.292495 1919 raft_consensus.cc:504] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.292886 1919 leader_election.cc:288] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [CANDIDATE]: Term 1 election: Requested vote from peers 1679b864b5d44183b9233c55f5913ef4 (127.13.148.1:43851), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
I20260806 16:06:43.293238 1459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca12ebcbf1fa4efeaa4ffa09fbb9cf44" candidate_uuid: "c106164208bf46c5b673c9fdca76e2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679b864b5d44183b9233c55f5913ef4"
I20260806 16:06:43.293344 1827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca12ebcbf1fa4efeaa4ffa09fbb9cf44" candidate_uuid: "c106164208bf46c5b673c9fdca76e2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5315bfa3b8414931999644babe6e6a48"
I20260806 16:06:43.293385 1459 raft_consensus.cc:3037] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:43.293521 1827 raft_consensus.cc:3037] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:43.294620 1459 raft_consensus.cc:2454] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c106164208bf46c5b673c9fdca76e2ae in term 1.
I20260806 16:06:43.294806 1827 raft_consensus.cc:2454] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c106164208bf46c5b673c9fdca76e2ae in term 1.
I20260806 16:06:43.294951 1552 leader_election.cc:302] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4, c106164208bf46c5b673c9fdca76e2ae; no voters:
I20260806 16:06:43.295215 1919 raft_consensus.cc:2781] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:06:43.295485 1919 raft_consensus.cc:686] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 1 LEADER]: Becoming Leader. State: Replica: c106164208bf46c5b673c9fdca76e2ae, State: Running, Role: LEADER
I20260806 16:06:43.295867 1919 consensus_queue.cc:237] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.299275 1232 catalog_manager.cc:5495] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae reported cstate change: term changed from 0 to 1, leader changed from <none> to c106164208bf46c5b673c9fdca76e2ae (127.13.148.2). New cstate: current_term: 1 leader_uuid: "c106164208bf46c5b673c9fdca76e2ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } health_report { overall_health: UNKNOWN } } }
I20260806 16:06:43.311743 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.308109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.13.148.2:43725)
0806 16:06:43.308264 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.308269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.308282 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:06:43.308391 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.308393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.308446 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.308538 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:06:43.308749 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.309234 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.310056 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.310265 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.310345 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.310796 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:06:43.311244 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:06:43.311338 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.311474 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.311527 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 16:06:43.317739 1459 raft_consensus.cc:1261] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Refusing update from remote peer c106164208bf46c5b673c9fdca76e2ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:06:43.317732 1827 raft_consensus.cc:1261] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 1 FOLLOWER]: Refusing update from remote peer c106164208bf46c5b673c9fdca76e2ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:06:43.318514 1919 consensus_queue.cc:1035] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:06:43.318810 1923 consensus_queue.cc:1035] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:06:43.326491 1929 mvcc.cc:205] Tried to move back new op lower bound from 7315588723982311424 to 7315588723902775296. Current Snapshot: MvccSnapshot[applied={T|T < 7315588723982311424}]
I20260806 16:06:43.335673 1238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:06:43.335844 1238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:06:43.338339 1238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Sending DeleteTablet for 3 replicas of tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44
I20260806 16:06:43.339128 1618 tablet_service.cc:1514] Processing DeleteTablet for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:06:43 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:54878
I20260806 16:06:43.339316 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:06:43 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:37398
I20260806 16:06:43.339493 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 1338
I20260806 16:06:43.343712 1942 tablet_replica.cc:330] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: stopping tablet replica
I20260806 16:06:43.344135 1940 tablet_replica.cc:330] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: stopping tablet replica
I20260806 16:06:43.344141 1942 raft_consensus.cc:2229] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:06:43.344457 1940 raft_consensus.cc:2229] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:06:43.344717 1942 raft_consensus.cc:2258] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:06:43.344906 1940 raft_consensus.cc:2258] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:06:43.347661 1940 ts_tablet_manager.cc:1905] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:06:43.347669 1942 ts_tablet_manager.cc:1905] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:06:43.350677 1942 ts_tablet_manager.cc:1918] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:06:43.350870 1942 log.cc:1196] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/ca12ebcbf1fa4efeaa4ffa09fbb9cf44
I20260806 16:06:43.351292 1942 ts_tablet_manager.cc:1939] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 5315bfa3b8414931999644babe6e6a48: Deleting consensus metadata
I20260806 16:06:43.351368 1940 ts_tablet_manager.cc:1918] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:06:43.351496 1940 log.cc:1196] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/ca12ebcbf1fa4efeaa4ffa09fbb9cf44
I20260806 16:06:43.351850 1940 ts_tablet_manager.cc:1939] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P c106164208bf46c5b673c9fdca76e2ae: Deleting consensus metadata
I20260806 16:06:43.352952 1220 catalog_manager.cc:4841] TS 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051): tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 (table test-table [id=e376bb7ec0954261ba031f0f3dbff6cb]) successfully deleted
I20260806 16:06:43.353008 1222 catalog_manager.cc:4841] TS c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725): tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 (table test-table [id=e376bb7ec0954261ba031f0f3dbff6cb]) successfully deleted
I20260806 16:06:43.354370 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 1527
W20260806 16:06:43.354377 1220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:43851 (error 108)
I20260806 16:06:43.360471 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.360054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43851 (local address 127.0.0.1:39480)
0806 16:06:43.360276 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.360356 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:43851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260806 16:06:43.360800 1220 catalog_manager.cc:4584] TS 1679b864b5d44183b9233c55f5913ef4 (127.13.148.1:43851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44: Network error: Client connection negotiation failed: client connection to 127.13.148.1:43851: connect: Connection refused (error 111)
I20260806 16:06:43.367442 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 1713
I20260806 16:06:43.381440 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 1194
I20260806 16:06:43.393880 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39593
--webserver_interface=127.13.148.62
--webserver_port=35227
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:43.508291 1944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:43.508651 1944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:43.508737 1944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:43.512841 1944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:43.512957 1944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:43.512998 1944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:43.513036 1944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:43.513073 1944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:43.518532 1944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:43.519688 1944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:43.520810 1944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:43.538087 1954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:43.538174 1953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:43.538241 1944 server_base.cc:1034] running on GCE node
W20260806 16:06:43.538524 1956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:43.539178 1944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:43.540370 1944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:43.541574 1944 hybrid_clock.cc:648] HybridClock initialized: now 1786032403541543 us; error 57 us; skew 500 ppm
Aug 06 16:06:43 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032403, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:43.544530 1944 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:06:43.545542 1944 webserver.cc:460] Webserver started at http://127.13.148.62:35227/ using document root <none> and password file <none>
I20260806 16:06:43.546169 1944 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:43.546289 1944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:43.550396 1944 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:06:43.552745 1968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:43.553996 1944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:06:43.554162 1944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "73a4743249f842fd980b83bee23698dd"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "ebc543aca44f27aa94793dcc923088cb"
server_key_iv: "345a31a860cf95ed79f1a9a845ddb107"
server_key_version: "encryptionkey@0"
I20260806 16:06:43.554883 1944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:43.572120 1944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:43.575696 1944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:43.575922 1944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:43.583562 1944 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39593
I20260806 16:06:43.583894 2041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39593 every 8 connection(s)
I20260806 16:06:43.584787 1944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:06:43.589422 2043 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:06:43.591215 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Bootstrap starting.
I20260806 16:06:43.591578 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 1944
I20260806 16:06:43.592041 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:43851
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:39593
--webserver_port=36327
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:06:43.608989 2043 log.cc:824] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:43.618309 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:06:43.618777 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Bootstrap complete.
I20260806 16:06:43.623101 2043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:43.623520 2043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a4743249f842fd980b83bee23698dd, State: Initialized, Role: FOLLOWER
I20260806 16:06:43.624034 2043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:43.624157 2043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:06:43.624254 2043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:06:43.624378 2043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:06:43.626389 2043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:43.626776 2043 leader_election.cc:302] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a4743249f842fd980b83bee23698dd; no voters:
I20260806 16:06:43.627133 2043 leader_election.cc:288] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 16:06:43.627223 2047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:06:43.627458 2047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 2 LEADER]: Becoming Leader. State: Replica: 73a4743249f842fd980b83bee23698dd, State: Running, Role: LEADER
I20260806 16:06:43.627768 2047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:43.628521 2043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:06:43.629181 2048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73a4743249f842fd980b83bee23698dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } } }
I20260806 16:06:43.629318 2048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:43.630077 2053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:06:43.630125 2049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73a4743249f842fd980b83bee23698dd. Latest consensus state: current_term: 2 leader_uuid: "73a4743249f842fd980b83bee23698dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } } }
I20260806 16:06:43.630251 2049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:43.635334 2053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:06:43.636055 2053 catalog_manager.cc:1259] Loaded cluster ID: 6ac3d370b8a64537b18ea6edeb4e26b3
I20260806 16:06:43.636126 2053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:06:43.637521 2053 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 16:06:43.642210 2053 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 16:06:43.647825 2053 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:06:43.648649 2053 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Loaded TSK: 0
I20260806 16:06:43.649901 2053 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:06:43.719084 2045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:43.719364 2045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:43.719431 2045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:43.723486 2045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:43.723552 2045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:43.723611 2045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:06:43.729022 2045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:43851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=36327
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:43.730209 2045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:43.731299 2045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:43.741147 2075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:43.741288 2045 server_base.cc:1034] running on GCE node
W20260806 16:06:43.741151 2078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:43.741500 2076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:43.741931 2045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:43.742784 2045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:43.743997 2045 hybrid_clock.cc:648] HybridClock initialized: now 1786032403743977 us; error 45 us; skew 500 ppm
Aug 06 16:06:43 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032403, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:43.747413 2045 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:06:43.748759 2045 webserver.cc:460] Webserver started at http://127.13.148.1:36327/ using document root <none> and password file <none>
I20260806 16:06:43.749624 2045 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:43.749758 2045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:43.755510 2045 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:06:43.758837 2089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:43.760423 2045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:06:43.760538 2045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "1679b864b5d44183b9233c55f5913ef4"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "748cf9220b2ae448c02ffe152d3ea109"
server_key_iv: "c6379de997add55f6c7ba02e43f26398"
server_key_version: "encryptionkey@0"
I20260806 16:06:43.761138 2045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:43.775326 2045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:43.778617 2045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:43.778831 2045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:43.779467 2045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:43.780629 2096 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 16:06:43.786404 2045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 16:06:43.786522 2045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 16:06:43.786612 2045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 16:06:43.788362 2045 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 16:06:43.788440 2045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 16:06:43.788718 2096 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Bootstrap starting.
I20260806 16:06:43.799332 2045 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:43851
I20260806 16:06:43.799391 2241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:43851 every 8 connection(s)
I20260806 16:06:43.800384 2045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:06:43.810758 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 2045
I20260806 16:06:43.810746 2096 log.cc:824] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:43.811260 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:43725
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:39593
--webserver_port=39265
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:06:43 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032403, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:43.819459 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.802760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:52881)
0806 16:06:43.803350 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.803405 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:43.804553 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:43.805788 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:43.805799 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:43.806286 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:43.807087 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.807115 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.808703 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.808707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.809964 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.809975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.810810 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.810814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.811006 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.811754 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:43.811781 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:43.813704 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:43.816318 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:43.816323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:43.816334 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:43.816594 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:43.817001 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:43.817005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:43.817007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:43.817151 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:43.817734 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:43.817740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:43.817977 (+ 237us) client_negotiation.cc:769] Sending connection context
0806 16:06:43.818166 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 16:06:43.818535 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":139,"threads_started":1}
I20260806 16:06:43.819445 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.802774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:52881 (local address 127.13.148.62:39593)
0806 16:06:43.803940 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.803988 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.804065 (+ 77us) server_negotiation.cc:407] Connection header received
0806 16:06:43.804989 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.805000 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.805316 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.805596 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:43.807319 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.808518 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.810096 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.810623 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.810993 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.813940 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:43.813947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:43.813965 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:43.814000 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:43.816190 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:43.816752 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:43.816766 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:43.816770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:43.816894 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:43.817252 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:43.817256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:43.817258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:43.817612 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:43.817763 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.818296 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.818597 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260806 16:06:43.820580 2096 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:06:43.821063 2096 tablet_bootstrap.cc:492] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Bootstrap complete.
I20260806 16:06:43.821748 2242 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39593
I20260806 16:06:43.822000 2242 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:43.822649 2096 ts_tablet_manager.cc:1397] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.002s
I20260806 16:06:43.822675 2242 heartbeater.cc:511] Master 127.13.148.62:39593 requested a full tablet report, sending...
I20260806 16:06:43.825479 1990 ts_manager.cc:194] Registered new tserver with Master: 1679b864b5d44183b9233c55f5913ef4 (127.13.148.1:43851)
I20260806 16:06:43.826581 2096 raft_consensus.cc:348] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.827253 2096 raft_consensus.cc:729] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679b864b5d44183b9233c55f5913ef4, State: Initialized, Role: FOLLOWER
I20260806 16:06:43.827769 2096 consensus_queue.cc:260] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } }
I20260806 16:06:43.828918 2096 ts_tablet_manager.cc:1428] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:06:43.830217 1990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:52881
I20260806 16:06:43.832989 2242 heartbeater.cc:503] Master 127.13.148.62:39593 was elected leader, sending a full tablet report...
Aug 06 16:06:43 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032403, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260806 16:06:43.840329 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.828581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43851 (local address 127.0.0.1:39496)
0806 16:06:43.829623 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:43.829682 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:43.829824 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:43.830509 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:43.830516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:43.830971 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:43.831104 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.831112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.832144 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.832147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.832768 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:43.832778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.833638 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.833642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:43.833765 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.834969 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:43.834980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:43.836394 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:43.838317 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:43.838327 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:43.838333 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:43.838672 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:43.839029 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:43.839032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:43.839034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:43.839128 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:43.839619 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:43.839625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:43.839768 (+ 143us) client_negotiation.cc:769] Sending connection context
0806 16:06:43.839910 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 16:06:43.840125 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":136,"threads_started":1}
I20260806 16:06:43.840329 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:43.828659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.13.148.1:43851)
0806 16:06:43.828955 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:43.828967 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:43.829792 (+ 825us) server_negotiation.cc:407] Connection header received
0806 16:06:43.829970 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:43.829975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:43.830187 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:43.830420 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:43.831247 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.832005 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.832894 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:43.833498 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:43.833628 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:43.836539 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:43.836549 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:43.836563 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:43.836595 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:43.838196 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:43.838826 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:43.838831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:43.838833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:43.838918 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:43.839234 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:43.839237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:43.839239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:43.839497 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:43.839648 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:43.840013 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 16:06:43.840183 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260806 16:06:43.841395 2154 tablet_service.cc:1514] Processing DeleteTablet for tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:06:43 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:39496
I20260806 16:06:43.842329 2253 tablet_replica.cc:330] stopping tablet replica
I20260806 16:06:43.842679 2253 raft_consensus.cc:2229] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:06:43.842947 2253 raft_consensus.cc:2258] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:06:43.843926 2253 ts_tablet_manager.cc:1905] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:06:43.847697 2253 ts_tablet_manager.cc:1918] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:06:43.847795 2253 log.cc:1196] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/ca12ebcbf1fa4efeaa4ffa09fbb9cf44
I20260806 16:06:43.848075 2253 ts_tablet_manager.cc:1939] T ca12ebcbf1fa4efeaa4ffa09fbb9cf44 P 1679b864b5d44183b9233c55f5913ef4: Deleting consensus metadata
I20260806 16:06:43.849815 1969 catalog_manager.cc:4841] TS 1679b864b5d44183b9233c55f5913ef4 (127.13.148.1:43851): tablet ca12ebcbf1fa4efeaa4ffa09fbb9cf44 (table test-table [id=e376bb7ec0954261ba031f0f3dbff6cb]) successfully deleted
W20260806 16:06:43.938102 2249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:43.938398 2249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:43.938529 2249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:43.942682 2249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:43.942768 2249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:43.942860 2249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:06:43.948153 2249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:43725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=39265
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:43.949229 2249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:43.950254 2249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:43.960317 2260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:43.960412 2249 server_base.cc:1034] running on GCE node
W20260806 16:06:43.960366 2259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:43.960317 2262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:43.961026 2249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:43.961686 2249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:43.962888 2249 hybrid_clock.cc:648] HybridClock initialized: now 1786032403962871 us; error 26 us; skew 500 ppm
Aug 06 16:06:43 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032403, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:43.966086 2249 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:06:43.967074 2249 webserver.cc:460] Webserver started at http://127.13.148.2:39265/ using document root <none> and password file <none>
I20260806 16:06:43.967700 2249 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:43.967825 2249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:43.971956 2249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:06:43.974012 2271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:43.975416 2249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:06:43.975571 2249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "c106164208bf46c5b673c9fdca76e2ae"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "bfaaf301fd9dce00c8408f39eed03eeb"
server_key_iv: "413a492584263b0e76ba04c9e78aaf3e"
server_key_version: "encryptionkey@0"
I20260806 16:06:43.976181 2249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:43.997027 2249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:44.000911 2249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:44.001174 2249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:44.001891 2249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:44.002846 2249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:44.002920 2249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:44.003000 2249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:44.003044 2249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:44.013108 2249 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:43725
I20260806 16:06:44.013139 2418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:43725 every 8 connection(s)
I20260806 16:06:44.014171 2249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:06:44.022289 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 2249
I20260806 16:06:44.022789 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:36051
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:39593
--webserver_port=42233
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:06:44 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032403, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:44.029990 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.016335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:36093 (local address 127.13.148.62:39593)
0806 16:06:44.016505 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.016509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.017351 (+ 842us) server_negotiation.cc:407] Connection header received
0806 16:06:44.019016 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.019020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.019095 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.019217 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:44.020536 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.021144 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.022536 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.022803 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.022937 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.025811 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:44.025814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:44.025817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:44.025845 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:44.027592 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:44.028174 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:44.028178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:44.028181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:44.028274 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:44.028677 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:44.028682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:44.028687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:44.028909 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:44.029030 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.029588 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.029802 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 16:06:44.031093 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.016688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:36093)
0806 16:06:44.017188 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:44.017234 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:44.018764 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:44.019418 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:44.019425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:44.019837 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:44.020361 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.020375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.021292 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.021301 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.022321 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.022332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.022982 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.022987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.023185 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.023867 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:44.023885 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:44.025612 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:44.027726 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:44.027733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:44.027751 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:44.028057 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:44.028381 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:44.028385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:44.028388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:44.028548 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:44.029029 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:44.029038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:44.029318 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 16:06:44.029529 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 16:06:44.029979 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260806 16:06:44.032492 2419 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39593
I20260806 16:06:44.032811 2419 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:44.033489 2419 heartbeater.cc:511] Master 127.13.148.62:39593 requested a full tablet report, sending...
I20260806 16:06:44.034962 1990 ts_manager.cc:194] Registered new tserver with Master: c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725)
I20260806 16:06:44.035902 1990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:36093
W20260806 16:06:44.142105 2423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:44.142369 2423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:44.142413 2423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:44.147095 2423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:44.147156 2423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:44.147215 2423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:06:44.152607 2423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:36051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=42233
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:44.153625 2423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:44.154688 2423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:44.164499 2433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:44.164587 2423 server_base.cc:1034] running on GCE node
W20260806 16:06:44.164526 2430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:44.164526 2431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:44.165289 2423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:44.165915 2423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:44.167128 2423 hybrid_clock.cc:648] HybridClock initialized: now 1786032404167095 us; error 42 us; skew 500 ppm
Aug 06 16:06:44 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032404, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:44.170114 2423 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:06:44.171116 2423 webserver.cc:460] Webserver started at http://127.13.148.3:42233/ using document root <none> and password file <none>
I20260806 16:06:44.171692 2423 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:44.171772 2423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:44.175698 2423 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 16:06:44.177888 2441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:44.179338 2423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:06:44.179472 2423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "5315bfa3b8414931999644babe6e6a48"
format_stamp: "Formatted at 2026-08-06 16:06:43 on dist-test-slave-7kzw"
server_key: "71c39efc9925b790fa90540fba0a1c2d"
server_key_iv: "e42deb42bde117f8fee6c695c3eb9391"
server_key_version: "encryptionkey@0"
I20260806 16:06:44.180048 2423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:44.201346 2423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:44.204874 2423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:44.205108 2423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:44.205749 2423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:44.206678 2423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:44.206751 2423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:44.206821 2423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:44.206872 2423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:44.216526 2423 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36051
I20260806 16:06:44.216622 2594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36051 every 8 connection(s)
I20260806 16:06:44.217602 2423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:06:44.221673 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 2423
Aug 06 16:06:44 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032404, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:06:44.232609 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.222411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.13.148.62:39593)
0806 16:06:44.222794 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.222797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.222807 (+ 10us) server_negotiation.cc:407] Connection header received
0806 16:06:44.223054 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.223057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.223100 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.223201 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:44.224247 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.225153 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.226510 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.226777 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.226877 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.228648 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:44.228651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:44.228653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:44.228727 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:44.230659 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:44.231255 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:44.231258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:44.231259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:44.231372 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:44.231655 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:44.231661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:44.231663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:44.231910 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:44.232141 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.232341 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.232437 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":322,"thread_start_us":127,"threads_started":1}
I20260806 16:06:44.234290 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.219718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:53635 (local address 127.13.148.62:39593)
0806 16:06:44.219863 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.219867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.220881 (+ 1014us) server_negotiation.cc:407] Connection header received
0806 16:06:44.222382 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.222386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.222444 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.222562 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:44.224382 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.225149 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.226782 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.227055 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.227128 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.230303 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:44.230309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:44.230312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:44.230355 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:44.231996 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:44.232643 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:44.232647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:44.232649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:44.232861 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:44.233198 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:44.233201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:44.233203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:44.233414 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:44.233534 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.234022 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.234138 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":53}
I20260806 16:06:44.235157 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.220131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:53635)
0806 16:06:44.220705 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:44.220751 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:44.222151 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:44.222860 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:44.222869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:44.223477 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:44.224214 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.224237 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.225311 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.225315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.226674 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.226685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.227221 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.227229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.227460 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.228073 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:44.228129 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:44.230139 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:44.232126 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:44.232133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:44.232149 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:44.232519 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:44.232976 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:44.232978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:44.232980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:44.233119 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:44.233550 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:44.233561 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:44.233790 (+ 229us) client_negotiation.cc:769] Sending connection context
0806 16:06:44.233951 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 16:06:44.234207 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":125,"threads_started":1}
I20260806 16:06:44.236326 2595 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39593
I20260806 16:06:44.236614 2595 heartbeater.cc:464] Registering TS with master...
I20260806 16:06:44.237147 2595 heartbeater.cc:511] Master 127.13.148.62:39593 requested a full tablet report, sending...
I20260806 16:06:44.238216 1990 ts_manager.cc:194] Registered new tserver with Master: 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
I20260806 16:06:44.238843 1990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:53635
I20260806 16:06:44.246512 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:06:44.251434 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.247190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.13.148.62:39593)
0806 16:06:44.247316 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.247320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.247516 (+ 196us) server_negotiation.cc:407] Connection header received
0806 16:06:44.247666 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.247668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.247764 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.247853 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:06:44.248249 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.248777 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.249694 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.249910 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.249997 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.250401 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:06:44.250985 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:06:44.251102 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.251269 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.251321 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:06:44.252168 1990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:06:44.255764 1990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:06:44.264397 2154 tablet_service.cc:1467] Processing CreateTablet for tablet f3772ea70c944e93a072c63dc24e4ce3 (DEFAULT_TABLE table=test-table [id=753eaf38bd954d6282eaa97f6af39acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:44.265187 2154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3772ea70c944e93a072c63dc24e4ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:44.268272 2096 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Bootstrap starting.
I20260806 16:06:44.269263 2096 tablet_bootstrap.cc:654] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:44.269333 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.263253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36051 (local address 127.0.0.1:51886)
0806 16:06:44.263951 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:44.263967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:44.264055 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:44.264826 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:44.264830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:44.264851 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:44.264974 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.264983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.266483 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.266490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.267385 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.267396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.268816 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.268821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.268913 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.268935 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:06:44.269012 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 16:06:44.269142 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":70,"thread_start_us":142,"threads_started":1}
I20260806 16:06:44.269593 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.263909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.13.148.3:36051)
0806 16:06:44.264317 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.264329 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.264362 (+ 33us) server_negotiation.cc:407] Connection header received
0806 16:06:44.264434 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.264437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.264640 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.265015 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:44.265285 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.266315 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.267537 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.268483 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.269115 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.269261 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.269316 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.269417 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":156,"threads_started":1}
I20260806 16:06:44.269606 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.263353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.13.148.2:43725)
0806 16:06:44.263682 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.263694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.263747 (+ 53us) server_negotiation.cc:407] Connection header received
0806 16:06:44.263831 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.263836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.264083 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.264255 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:44.264865 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.265929 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.267564 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.268426 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.268529 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.268680 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.269061 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.269176 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":146,"threads_started":1}
I20260806 16:06:44.269734 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.263184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:43725 (local address 127.0.0.1:39996)
0806 16:06:44.263543 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:44.263561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:44.263673 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:44.264465 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:44.264468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:44.264487 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:44.264589 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.264599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.266106 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.266109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.267421 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.267433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.268749 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.268753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.268850 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.268900 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 16:06:44.268985 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 16:06:44.269611 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":36}
I20260806 16:06:44.270925 2096 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: No bootstrap required, opened a new log
I20260806 16:06:44.271041 2096 ts_tablet_manager.cc:1397] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 16:06:44.271665 2096 raft_consensus.cc:348] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.271806 2096 raft_consensus.cc:374] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:44.271858 2096 raft_consensus.cc:729] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679b864b5d44183b9233c55f5913ef4, State: Initialized, Role: FOLLOWER
I20260806 16:06:44.272042 2096 consensus_queue.cc:260] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.272336 2096 ts_tablet_manager.cc:1428] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 16:06:44.272828 2330 tablet_service.cc:1467] Processing CreateTablet for tablet f3772ea70c944e93a072c63dc24e4ce3 (DEFAULT_TABLE table=test-table [id=753eaf38bd954d6282eaa97f6af39acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:44.272828 2502 tablet_service.cc:1467] Processing CreateTablet for tablet f3772ea70c944e93a072c63dc24e4ce3 (DEFAULT_TABLE table=test-table [id=753eaf38bd954d6282eaa97f6af39acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:06:44.273972 2330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3772ea70c944e93a072c63dc24e4ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:44.273976 2502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3772ea70c944e93a072c63dc24e4ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:06:44.282246 2614 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Bootstrap starting.
I20260806 16:06:44.283108 2612 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: Bootstrap starting.
I20260806 16:06:44.284745 2614 tablet_bootstrap.cc:654] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:44.285547 2612 tablet_bootstrap.cc:654] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: Neither blocks nor log segments found. Creating new log.
I20260806 16:06:44.285974 2614 log.cc:824] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:44.286301 2612 log.cc:824] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:44.288339 2614 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: No bootstrap required, opened a new log
I20260806 16:06:44.288335 2612 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: No bootstrap required, opened a new log
I20260806 16:06:44.288551 2612 ts_tablet_manager.cc:1397] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:06:44.288551 2614 ts_tablet_manager.cc:1397] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:06:44.292615 2614 raft_consensus.cc:348] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.292615 2612 raft_consensus.cc:348] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.292917 2614 raft_consensus.cc:374] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:44.292917 2612 raft_consensus.cc:374] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:06:44.292972 2614 raft_consensus.cc:729] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5315bfa3b8414931999644babe6e6a48, State: Initialized, Role: FOLLOWER
I20260806 16:06:44.292984 2612 raft_consensus.cc:729] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c106164208bf46c5b673c9fdca76e2ae, State: Initialized, Role: FOLLOWER
I20260806 16:06:44.293468 2614 consensus_queue.cc:260] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.293468 2612 consensus_queue.cc:260] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.294237 2595 heartbeater.cc:503] Master 127.13.148.62:39593 was elected leader, sending a full tablet report...
I20260806 16:06:44.294238 2419 heartbeater.cc:503] Master 127.13.148.62:39593 was elected leader, sending a full tablet report...
I20260806 16:06:44.294590 2614 ts_tablet_manager.cc:1428] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 16:06:44.294590 2612 ts_tablet_manager.cc:1428] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
W20260806 16:06:44.301585 2243 tablet.cc:2369] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:06:44.434813 2250 raft_consensus.cc:482] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:44.435179 2250 raft_consensus.cc:504] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.436403 2250 leader_election.cc:288] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
I20260806 16:06:44.440603 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.436836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42315 (local address 127.13.148.2:43725)
0806 16:06:44.436971 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.436977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.437315 (+ 338us) server_negotiation.cc:407] Connection header received
0806 16:06:44.437502 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.437507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.437560 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.437652 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:44.437921 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.438421 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.439596 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.439994 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.440130 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.440188 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.440368 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.440439 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:06:44.440615 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.436832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:60341 (local address 127.13.148.3:36051)
0806 16:06:44.437077 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.437082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.437278 (+ 196us) server_negotiation.cc:407] Connection header received
0806 16:06:44.437502 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.437505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.437552 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.437646 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:06:44.437952 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.438441 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.439640 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.440039 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.440178 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.440227 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.440420 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.440485 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260806 16:06:44.440670 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.436689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:43725 (local address 127.13.148.1:42315)
0806 16:06:44.437042 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:44.437056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:44.437372 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:44.437666 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:44.437669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:44.437700 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:44.437808 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.437813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.438599 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.438603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.439468 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.439477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.440116 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.440120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.440208 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.440260 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 16:06:44.440351 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 16:06:44.440431 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":3,"spinlock_wait_cycles":11136,"thread_start_us":96,"threads_started":1}
I20260806 16:06:44.440676 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.436694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36051 (local address 127.13.148.1:60341)
0806 16:06:44.437106 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:44.437169 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:44.437355 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:44.437662 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:44.437664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:44.437684 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:06:44.437830 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.437835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.438591 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.438594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.439479 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:44.439489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.440173 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.440176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:44.440275 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.440297 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 16:06:44.440392 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 16:06:44.440468 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":110,"spinlock_wait_cycles":22400,"thread_start_us":87,"threads_started":1}
I20260806 16:06:44.441339 2360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3772ea70c944e93a072c63dc24e4ce3" candidate_uuid: "1679b864b5d44183b9233c55f5913ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106164208bf46c5b673c9fdca76e2ae" is_pre_election: true
I20260806 16:06:44.441614 2534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3772ea70c944e93a072c63dc24e4ce3" candidate_uuid: "1679b864b5d44183b9233c55f5913ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5315bfa3b8414931999644babe6e6a48" is_pre_election: true
I20260806 16:06:44.441702 2360 raft_consensus.cc:2454] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679b864b5d44183b9233c55f5913ef4 in term 0.
I20260806 16:06:44.441898 2534 raft_consensus.cc:2454] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679b864b5d44183b9233c55f5913ef4 in term 0.
I20260806 16:06:44.442328 2092 leader_election.cc:302] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4, 5315bfa3b8414931999644babe6e6a48, c106164208bf46c5b673c9fdca76e2ae; no voters:
I20260806 16:06:44.442672 2250 raft_consensus.cc:2781] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:06:44.442768 2250 raft_consensus.cc:482] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:06:44.442816 2250 raft_consensus.cc:3037] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:44.443864 2250 raft_consensus.cc:504] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.444222 2250 leader_election.cc:288] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 1 election: Requested vote from peers c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
I20260806 16:06:44.444586 2360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3772ea70c944e93a072c63dc24e4ce3" candidate_uuid: "1679b864b5d44183b9233c55f5913ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106164208bf46c5b673c9fdca76e2ae"
I20260806 16:06:44.444607 2534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3772ea70c944e93a072c63dc24e4ce3" candidate_uuid: "1679b864b5d44183b9233c55f5913ef4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5315bfa3b8414931999644babe6e6a48"
I20260806 16:06:44.444733 2534 raft_consensus.cc:3037] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:44.444734 2360 raft_consensus.cc:3037] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:06:44.446084 2534 raft_consensus.cc:2454] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679b864b5d44183b9233c55f5913ef4 in term 1.
I20260806 16:06:44.446084 2360 raft_consensus.cc:2454] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679b864b5d44183b9233c55f5913ef4 in term 1.
I20260806 16:06:44.446393 2090 leader_election.cc:302] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4, 5315bfa3b8414931999644babe6e6a48, c106164208bf46c5b673c9fdca76e2ae; no voters:
I20260806 16:06:44.446606 2250 raft_consensus.cc:2781] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:06:44.446833 2250 raft_consensus.cc:686] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 LEADER]: Becoming Leader. State: Replica: 1679b864b5d44183b9233c55f5913ef4, State: Running, Role: LEADER
I20260806 16:06:44.447101 2250 consensus_queue.cc:237] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:44.450035 1990 catalog_manager.cc:5495] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679b864b5d44183b9233c55f5913ef4 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "1679b864b5d44183b9233c55f5913ef4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } health_report { overall_health: HEALTHY } } }
I20260806 16:06:44.465246 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.461242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.13.148.1:43851)
0806 16:06:44.461526 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:44.461533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:44.461546 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:06:44.461623 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:44.461625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:44.461676 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:44.461779 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:06:44.462049 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.462664 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.463510 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:44.463824 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:44.463924 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:44.464242 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:06:44.464730 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:06:44.464823 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:44.464979 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 16:06:44.465045 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":123,"threads_started":1}
W20260806 16:06:44.468606 2596 tablet.cc:2369] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:06:44.469620 2360 raft_consensus.cc:1261] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 1 FOLLOWER]: Refusing update from remote peer 1679b864b5d44183b9233c55f5913ef4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:06:44.469619 2534 raft_consensus.cc:1261] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 1 FOLLOWER]: Refusing update from remote peer 1679b864b5d44183b9233c55f5913ef4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:06:44.470404 2624 consensus_queue.cc:1035] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:06:44.470705 2250 consensus_queue.cc:1035] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:06:44.480531 2631 mvcc.cc:205] Tried to move back new op lower bound from 7315588728700211200 to 7315588728617836544. Current Snapshot: MvccSnapshot[applied={T|T < 7315588728700211200}]
I20260806 16:06:44.480768 2630 mvcc.cc:205] Tried to move back new op lower bound from 7315588728700211200 to 7315588728617836544. Current Snapshot: MvccSnapshot[applied={T|T < 7315588728700211200}]
I20260806 16:06:44.486810 1989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:06:44.486965 1989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:06:44.489341 1989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Sending DeleteTablet for 3 replicas of tablet f3772ea70c944e93a072c63dc24e4ce3
I20260806 16:06:44.490156 2330 tablet_service.cc:1514] Processing DeleteTablet for tablet f3772ea70c944e93a072c63dc24e4ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:06:44 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:39996
I20260806 16:06:44.490175 2502 tablet_service.cc:1514] Processing DeleteTablet for tablet f3772ea70c944e93a072c63dc24e4ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:06:44 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260806 16:06:44.490401 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 2045
I20260806 16:06:44.496004 2638 tablet_replica.cc:330] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: stopping tablet replica
I20260806 16:06:44.496368 2638 raft_consensus.cc:2229] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:06:44.496690 2638 raft_consensus.cc:2258] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:06:44.497076 2639 tablet_replica.cc:330] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: stopping tablet replica
I20260806 16:06:44.497401 2639 raft_consensus.cc:2229] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:06:44.497700 2639 raft_consensus.cc:2258] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:06:44.499114 2638 ts_tablet_manager.cc:1905] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:06:44.500371 2639 ts_tablet_manager.cc:1905] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:06:44.502796 2639 ts_tablet_manager.cc:1918] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:06:44.502952 2639 log.cc:1196] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/f3772ea70c944e93a072c63dc24e4ce3
I20260806 16:06:44.503278 2639 ts_tablet_manager.cc:1939] T f3772ea70c944e93a072c63dc24e4ce3 P c106164208bf46c5b673c9fdca76e2ae: Deleting consensus metadata
I20260806 16:06:44.504244 2638 ts_tablet_manager.cc:1918] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:06:44.504412 2638 log.cc:1196] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/f3772ea70c944e93a072c63dc24e4ce3
I20260806 16:06:44.504557 1972 catalog_manager.cc:4841] TS c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725): tablet f3772ea70c944e93a072c63dc24e4ce3 (table test-table [id=753eaf38bd954d6282eaa97f6af39acd]) successfully deleted
I20260806 16:06:44.504853 2638 ts_tablet_manager.cc:1939] T f3772ea70c944e93a072c63dc24e4ce3 P 5315bfa3b8414931999644babe6e6a48: Deleting consensus metadata
I20260806 16:06:44.506228 1969 catalog_manager.cc:4841] TS 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051): tablet f3772ea70c944e93a072c63dc24e4ce3 (table test-table [id=753eaf38bd954d6282eaa97f6af39acd]) successfully deleted
I20260806 16:06:44.509682 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 2249
W20260806 16:06:44.509665 1969 connection.cc:535] client connection to 127.13.148.1:43851 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 16:06:44.509873 1969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 16:06:44.516886 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:44.516577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43851 (local address 127.0.0.1:39518)
0806 16:06:44.516731 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:44.516802 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:43851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 16:06:44.517127 1969 catalog_manager.cc:4584] TS 1679b864b5d44183b9233c55f5913ef4 (127.13.148.1:43851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3772ea70c944e93a072c63dc24e4ce3: Network error: Client connection negotiation failed: client connection to 127.13.148.1:43851: connect: Connection refused (error 111)
I20260806 16:06:44.522186 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 2423
I20260806 16:06:44.534760 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 1944
I20260806 16:06:44.547585 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39593
--webserver_interface=127.13.148.62
--webserver_port=35227
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:06:44.665196 2641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:44.665534 2641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:44.665616 2641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:44.670434 2641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:06:44.670550 2641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:44.670588 2641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:44.670624 2641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:06:44.670655 2641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:06:44.676123 2641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:44.677263 2641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:44.678323 2641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:44.687369 2647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:44.687369 2646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:44.687489 2641 server_base.cc:1034] running on GCE node
W20260806 16:06:44.687373 2650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:44.688472 2641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:44.689735 2641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:44.690946 2641 hybrid_clock.cc:648] HybridClock initialized: now 1786032404690918 us; error 50 us; skew 500 ppm
Aug 06 16:06:44 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032404, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:44.693928 2641 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:06:44.694975 2641 webserver.cc:460] Webserver started at http://127.13.148.62:35227/ using document root <none> and password file <none>
I20260806 16:06:44.695544 2641 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:44.695648 2641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:44.699620 2641 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:06:44.701973 2657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:44.703341 2641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:06:44.703502 2641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "73a4743249f842fd980b83bee23698dd"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "ebc543aca44f27aa94793dcc923088cb"
server_key_iv: "345a31a860cf95ed79f1a9a845ddb107"
server_key_version: "encryptionkey@0"
I20260806 16:06:44.704074 2641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:44.728569 2641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:44.732149 2641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:44.732434 2641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:44.740330 2641 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39593
I20260806 16:06:44.740347 2733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39593 every 8 connection(s)
I20260806 16:06:44.741474 2641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:06:44.745713 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 2641
I20260806 16:06:44.746037 2734 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:06:44.746255 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:43851
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:39593
--webserver_port=36327
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:06:44.748009 2734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Bootstrap starting.
I20260806 16:06:44.766582 2734 log.cc:824] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:44.778053 2734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:06:44.778568 2734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd: Bootstrap complete.
I20260806 16:06:44.783241 2734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:44.783695 2734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a4743249f842fd980b83bee23698dd, State: Initialized, Role: FOLLOWER
I20260806 16:06:44.784235 2734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:44.784382 2734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:06:44.784472 2734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:06:44.784590 2734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 2 FOLLOWER]: Advancing to term 3
I20260806 16:06:44.786872 2734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:44.787236 2734 leader_election.cc:302] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a4743249f842fd980b83bee23698dd; no voters:
I20260806 16:06:44.787639 2734 leader_election.cc:288] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 16:06:44.787729 2740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 3 FOLLOWER]: Leader election won for term 3
I20260806 16:06:44.787930 2740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [term 3 LEADER]: Becoming Leader. State: Replica: 73a4743249f842fd980b83bee23698dd, State: Running, Role: LEADER
I20260806 16:06:44.788239 2740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } }
I20260806 16:06:44.788937 2734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:06:44.789198 2740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73a4743249f842fd980b83bee23698dd. Latest consensus state: current_term: 3 leader_uuid: "73a4743249f842fd980b83bee23698dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } } }
I20260806 16:06:44.789319 2740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:44.789690 2744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:06:44.789728 2741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "73a4743249f842fd980b83bee23698dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a4743249f842fd980b83bee23698dd" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39593 } } }
I20260806 16:06:44.789953 2741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a4743249f842fd980b83bee23698dd [sys.catalog]: This master's current role is: LEADER
I20260806 16:06:44.795089 2744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:06:44.795900 2744 catalog_manager.cc:1259] Loaded cluster ID: 6ac3d370b8a64537b18ea6edeb4e26b3
I20260806 16:06:44.795967 2744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 16:06:44.797040 2744 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3c06e689ad google::LogMessage::Fail() at ??:0
@ 0x7f3c06e6b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3c06e684de google::LogMessage::Flush() at ??:0
@ 0x7f3c06e69a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3c117abfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3c117acd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3c117a9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3c117ed640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3c090ba3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3c090bac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3c090bc3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3c090ab512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3c0a6ef6db start_thread at ??:0
@ 0x7f3c0596661f clone at ??:0
W20260806 16:06:44.892879 2738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:44.893172 2738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:44.893222 2738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:44.898278 2738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:44.898361 2738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:44.898427 2738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:06:44.904260 2738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:43851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=36327
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:44.905501 2738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:44.906821 2738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:06:44.917299 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:44.917289 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:44.917308 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:44.917698 2738 server_base.cc:1034] running on GCE node
I20260806 16:06:44.918177 2738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:44.918918 2738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:44.920109 2738 hybrid_clock.cc:648] HybridClock initialized: now 1786032404920074 us; error 54 us; skew 500 ppm
Aug 06 16:06:44 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032404, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:44.923237 2738 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:06:44.924225 2738 webserver.cc:460] Webserver started at http://127.13.148.1:36327/ using document root <none> and password file <none>
I20260806 16:06:44.924887 2738 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:44.924985 2738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:44.929387 2738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:06:44.932046 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:44.933554 2738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:06:44.933679 2738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "1679b864b5d44183b9233c55f5913ef4"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "748cf9220b2ae448c02ffe152d3ea109"
server_key_iv: "c6379de997add55f6c7ba02e43f26398"
server_key_version: "encryptionkey@0"
I20260806 16:06:44.934327 2738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:44.963950 2738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:44.967892 2738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:44.968127 2738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:44.968853 2738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:44.970196 2813 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 16:06:44.976320 2738 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 16:06:44.976454 2738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260806 16:06:44.976594 2738 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 16:06:44.978438 2738 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 16:06:44.978585 2738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260806 16:06:44.978906 2813 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Bootstrap starting.
I20260806 16:06:44.990078 2738 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:43851
I20260806 16:06:44.990969 2958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:43851 every 8 connection(s)
I20260806 16:06:44.991530 2738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:06:44.995798 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 2738
I20260806 16:06:44.996258 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:43725
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:39593
--webserver_port=39265
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:06:45.005226 2813 log.cc:824] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:06:45.013798 2813 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:06:45.014261 2813 tablet_bootstrap.cc:492] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Bootstrap complete.
I20260806 16:06:45.015595 2813 ts_tablet_manager.cc:1397] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.007s
I20260806 16:06:45.018242 2813 raft_consensus.cc:348] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:45.018813 2813 raft_consensus.cc:729] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679b864b5d44183b9233c55f5913ef4, State: Initialized, Role: FOLLOWER
I20260806 16:06:45.019357 2813 consensus_queue.cc:260] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:45.020449 2813 ts_tablet_manager.cc:1428] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260806 16:06:45.048758 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:06:44.993639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:42753)
0806 16:06:44.994145 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:44.994201 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:44.995091 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:45.047922 (+ 52831us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":148,"threads_started":1}
W20260806 16:06:45.049474 2959 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:39593 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:39593: Client connection negotiation failed: client connection to 127.13.148.62:39593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 16:06:45.050438 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:45.050159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:47057)
0806 16:06:45.050294 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:45.050375 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 16:06:45.051407 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:45.051097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:50537)
0806 16:06:45.051290 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:45.051342 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260806 16:06:45.051815 2959 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:06:45.137985 2962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:45.138330 2962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:45.138409 2962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:45.142657 2962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:45.142731 2962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:45.142830 2962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:06:45.148273 2962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:43725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=39265
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:45.149390 2962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:45.150498 2962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:06:45.160661 2962 server_base.cc:1034] running on GCE node
W20260806 16:06:45.160647 2974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:45.160629 2972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:45.160933 2978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:45.161415 2962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:45.162087 2962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:45.163309 2962 hybrid_clock.cc:648] HybridClock initialized: now 1786032405163289 us; error 29 us; skew 500 ppm
Aug 06 16:06:45 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032405, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:45.166213 2962 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:06:45.167181 2962 webserver.cc:460] Webserver started at http://127.13.148.2:39265/ using document root <none> and password file <none>
I20260806 16:06:45.167841 2962 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:45.167940 2962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:45.171947 2962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:06:45.174244 2990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:45.175824 2962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:06:45.175989 2962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "c106164208bf46c5b673c9fdca76e2ae"
format_stamp: "Formatted at 2026-08-06 16:06:42 on dist-test-slave-7kzw"
server_key: "bfaaf301fd9dce00c8408f39eed03eeb"
server_key_iv: "413a492584263b0e76ba04c9e78aaf3e"
server_key_version: "encryptionkey@0"
I20260806 16:06:45.176589 2962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:45.190209 2962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:45.193638 2962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:45.193873 2962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:45.194581 2962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:45.195493 2962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:45.195569 2962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:45.195650 2962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:45.195694 2962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:45.205368 2962 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:43725
I20260806 16:06:45.205446 3144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:43725 every 8 connection(s)
I20260806 16:06:45.206514 2962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:06:45.209206 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 2962
I20260806 16:06:45.209873 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:36051
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:39593
--webserver_port=42233
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:06:45.211262 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:45.208963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:53239)
0806 16:06:45.209605 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:45.209724 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":165,"threads_started":1}
W20260806 16:06:45.212103 3146 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:39593 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:39593: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
I20260806 16:06:45.212985 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:45.212658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:47301)
0806 16:06:45.212830 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:45.212903 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 16:06:45.213888 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:45.213601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:57001)
0806 16:06:45.213740 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:45.213803 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 16:06:45.214175 3146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:06:45.325148 3150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:06:45.325412 3150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:06:45.325453 3150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:06:45.329473 3150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:06:45.329540 3150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:06:45.329627 3150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:06:45.335072 3150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:36051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=42233
--tserver_master_addrs=127.13.148.62:39593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:06:45.336092 3150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:06:45.337116 3150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:06:45.347522 3150 server_base.cc:1034] running on GCE node
W20260806 16:06:45.347606 3156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:45.347710 3158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:06:45.347769 3160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:06:45.348251 3150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:06:45.348981 3150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:06:45.350214 3150 hybrid_clock.cc:648] HybridClock initialized: now 1786032405350175 us; error 49 us; skew 500 ppm
Aug 06 16:06:45 dist-test-slave-7kzw krb5kdc[1172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032405, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:06:45.353421 3150 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:06:45.354404 3150 webserver.cc:460] Webserver started at http://127.13.148.3:42233/ using document root <none> and password file <none>
I20260806 16:06:45.355014 3150 fs_manager.cc:359] Metadata directory not provided
I20260806 16:06:45.355101 3150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:06:45.359045 3150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:06:45.361186 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:45.362682 3150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:06:45.362797 3150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "5315bfa3b8414931999644babe6e6a48"
format_stamp: "Formatted at 2026-08-06 16:06:43 on dist-test-slave-7kzw"
server_key: "71c39efc9925b790fa90540fba0a1c2d"
server_key_iv: "e42deb42bde117f8fee6c695c3eb9391"
server_key_version: "encryptionkey@0"
I20260806 16:06:45.363328 3150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:06:45.396672 3150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:06:45.400189 3150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:06:45.400353 3150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:06:45.400919 3150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:06:45.401726 3150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:06:45.401774 3150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:45.401815 3150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:06:45.401837 3150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:06:45.411298 3150 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36051
I20260806 16:06:45.411320 3324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36051 every 8 connection(s)
I20260806 16:06:45.412386 3150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:06:45.416393 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:45.414670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:38165)
0806 16:06:45.415125 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:45.415242 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
W20260806 16:06:45.416903 3325 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:39593 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:39593: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
I20260806 16:06:45.417470 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:45.417264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:45141)
0806 16:06:45.417369 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:45.417416 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 16:06:45.418107 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:45.417910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:38135)
0806 16:06:45.418011 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:45.418062 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:06:45.418274 3325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 16:06:45.419009 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 3150
I20260806 16:06:46.053038 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:46.052543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:44451)
0806 16:06:46.052843 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:46.052912 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":88,"threads_started":1}
I20260806 16:06:46.215319 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:46.214856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:35315)
0806 16:06:46.215125 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:46.215205 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
I20260806 16:06:46.419425 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:46.418933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:41017)
0806 16:06:46.419239 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:46.419304 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20260806 16:06:46.492417 3333 raft_consensus.cc:482] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:46.492624 3333 raft_consensus.cc:504] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:46.493726 3333 leader_election.cc:288] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
Aug 06 16:06:46 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032404, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.3@KRBTEST.COM
Aug 06 16:06:46 dist-test-slave-7kzw krb5kdc[1172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032404, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.2@KRBTEST.COM
I20260806 16:06:46.516139 3338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:46.494158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:59503 (local address 127.13.148.3:36051)
0806 16:06:46.494597 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:46.494626 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:46.494676 (+ 50us) server_negotiation.cc:407] Connection header received
0806 16:06:46.495586 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:46.495594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:46.496046 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:46.496389 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:46.498276 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:46.499928 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:46.501819 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:46.502568 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:46.502758 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:46.510008 (+ 7250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:46.510013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:46.510028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:46.510088 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:46.512340 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:46.513215 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:46.513222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:46.513228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:46.513412 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:46.513983 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:46.513987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:46.513989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:46.514603 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:46.514773 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:46.515706 (+ 933us) server_negotiation.cc:299] Negotiation successful
0806 16:06:46.515953 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":88,"threads_started":1}
I20260806 16:06:46.516238 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:46.493977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:43725 (local address 127.13.148.1:39311)
0806 16:06:46.494425 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:46.494440 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:46.494547 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:46.496499 (+ 1952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:46.496505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:46.497164 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:46.498048 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:46.498070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:46.500100 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:46.500104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:46.501684 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:46.501691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:46.502708 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:46.502712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:46.503085 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:46.507631 (+ 4546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:46.507668 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:46.509992 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:46.512509 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:46.512516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:46.512533 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:46.513074 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:46.513571 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:46.513576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:46.513578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:46.513865 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:46.514813 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:46.514821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:46.515241 (+ 420us) client_negotiation.cc:769] Sending connection context
0806 16:06:46.515515 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 16:06:46.515986 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":425,"spinlock_wait_cycles":50944,"thread_start_us":101,"threads_started":1}
I20260806 16:06:46.516238 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:46.493977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36051 (local address 127.13.148.1:59503)
0806 16:06:46.494232 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:46.494249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:06:46.494359 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:06:46.496499 (+ 2140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:06:46.496506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:06:46.497164 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:06:46.498048 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:46.498070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:46.500100 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:46.500104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:46.501679 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:06:46.501691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:06:46.502708 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:06:46.502712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:06:46.503089 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:46.507631 (+ 4542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:06:46.507657 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:06:46.509749 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:06:46.512509 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:46.512516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:46.512533 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:46.513074 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:46.513571 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:06:46.513576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:06:46.513578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:06:46.513865 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:06:46.514813 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:06:46.514821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:06:46.515241 (+ 420us) client_negotiation.cc:769] Sending connection context
0806 16:06:46.515511 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 16:06:46.516047 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":675,"spinlock_wait_cycles":33792}
I20260806 16:06:46.517514 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:46.494185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:39311 (local address 127.13.148.2:43725)
0806 16:06:46.494597 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0806 16:06:46.494626 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 16:06:46.494675 (+ 49us) server_negotiation.cc:407] Connection header received
0806 16:06:46.495586 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:06:46.495594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:06:46.496046 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:06:46.496389 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:06:46.498276 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:46.499928 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:46.501819 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:06:46.502568 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:06:46.502739 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:06:46.510205 (+ 7466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:06:46.510214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:06:46.510232 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:06:46.510266 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:06:46.512340 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:46.513224 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:46.513229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:46.513233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:46.513412 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:06:46.513983 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:06:46.513987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:06:46.513989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:06:46.514603 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:06:46.514748 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 16:06:46.516794 (+ 2046us) server_negotiation.cc:299] Negotiation successful
0806 16:06:46.517038 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":62,"threads_started":1}
W20260806 16:06:46.519297 2803 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
W20260806 16:06:46.520027 2806 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
I20260806 16:06:46.520120 2806 leader_election.cc:302] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4; no voters: 5315bfa3b8414931999644babe6e6a48, c106164208bf46c5b673c9fdca76e2ae
I20260806 16:06:46.520398 3333 raft_consensus.cc:2726] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:06:47.054584 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:47.054012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:49941)
0806 16:06:47.054364 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:47.054421 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":99,"threads_started":1}
I20260806 16:06:47.216678 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:47.216202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:58829)
0806 16:06:47.216514 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:47.216576 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260806 16:06:47.420748 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:47.420286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:43595)
0806 16:06:47.420595 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:47.420651 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
I20260806 16:06:48.055907 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:48.055439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:49641)
0806 16:06:48.055731 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:48.055800 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260806 16:06:48.076818 3346 raft_consensus.cc:482] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:48.076992 3346 raft_consensus.cc:504] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:48.077482 3346 leader_election.cc:288] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
W20260806 16:06:48.078348 2803 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
W20260806 16:06:48.078517 2806 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
I20260806 16:06:48.078612 2806 leader_election.cc:302] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4; no voters: 5315bfa3b8414931999644babe6e6a48, c106164208bf46c5b673c9fdca76e2ae
I20260806 16:06:48.078807 3346 raft_consensus.cc:2726] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:06:48.217849 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:48.217442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:37275)
0806 16:06:48.217698 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:48.217753 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260806 16:06:48.422009 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:48.421553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:43547)
0806 16:06:48.421848 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:48.421912 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260806 16:06:49.057356 3351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:49.056902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:54669)
0806 16:06:49.057193 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:49.057254 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260806 16:06:49.219064 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:49.218622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:51923)
0806 16:06:49.218911 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:49.218967 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":74,"threads_started":1}
I20260806 16:06:49.423285 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:49.422870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:43007)
0806 16:06:49.423132 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:49.423191 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260806 16:06:50.059155 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:50.058644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:55603)
0806 16:06:50.058985 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:50.059050 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":82,"threads_started":1}
I20260806 16:06:50.220427 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:50.219943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:51177)
0806 16:06:50.220246 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:50.220318 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260806 16:06:50.424659 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:50.424201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:50931)
0806 16:06:50.424481 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:50.424555 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260806 16:06:51.060783 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:51.060257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:56797)
0806 16:06:51.060568 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:51.060673 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260806 16:06:51.221876 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:51.221386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:37469)
0806 16:06:51.221697 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:51.221764 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260806 16:06:51.276634 3363 raft_consensus.cc:482] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:51.276793 3363 raft_consensus.cc:504] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:51.277292 3363 leader_election.cc:288] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
W20260806 16:06:51.278177 2803 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
W20260806 16:06:51.278278 2806 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
I20260806 16:06:51.278409 2806 leader_election.cc:302] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4; no voters: 5315bfa3b8414931999644babe6e6a48, c106164208bf46c5b673c9fdca76e2ae
I20260806 16:06:51.278642 3363 raft_consensus.cc:2726] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:06:51.425976 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:51.425550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:59101)
0806 16:06:51.425825 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:51.425877 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260806 16:06:52.062168 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:52.061729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:39023)
0806 16:06:52.062006 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:52.062071 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260806 16:06:52.223531 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:52.222936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:44519)
0806 16:06:52.223342 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:52.223423 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":98,"threads_started":1}
I20260806 16:06:52.427431 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:52.426939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:53033)
0806 16:06:52.427261 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:52.427331 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":91,"threads_started":1}
I20260806 16:06:52.782279 3371 raft_consensus.cc:482] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:52.782425 3371 raft_consensus.cc:504] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:52.782922 3371 leader_election.cc:288] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
W20260806 16:06:52.783758 2806 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
W20260806 16:06:52.783849 2803 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
I20260806 16:06:52.783986 2803 leader_election.cc:302] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4; no voters: 5315bfa3b8414931999644babe6e6a48, c106164208bf46c5b673c9fdca76e2ae
I20260806 16:06:52.784127 3371 raft_consensus.cc:2726] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:06:53.063457 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:53.063015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:38219)
0806 16:06:53.063287 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:53.063357 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":126,"threads_started":1}
I20260806 16:06:53.224793 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:53.224337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:54453)
0806 16:06:53.224614 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:53.224697 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
I20260806 16:06:53.428741 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:53.428256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:51587)
0806 16:06:53.428533 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:53.428595 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260806 16:06:54.064849 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:54.064360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:46673)
0806 16:06:54.064673 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:54.064743 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260806 16:06:54.226071 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:54.225642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:56301)
0806 16:06:54.225909 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:54.225970 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260806 16:06:54.429934 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:54.429548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:58577)
0806 16:06:54.429781 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:54.429835 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260806 16:06:55.066412 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:55.065898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:58783)
0806 16:06:55.066177 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:55.066294 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":113,"threads_started":1}
I20260806 16:06:55.227484 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:55.227042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:33495)
0806 16:06:55.227325 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:55.227383 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260806 16:06:55.431247 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:55.430818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:49351)
0806 16:06:55.431079 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:55.431148 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260806 16:06:56.067780 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:56.067249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:35571)
0806 16:06:56.067536 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:56.067595 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260806 16:06:56.228693 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:56.228287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:45801)
0806 16:06:56.228533 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:56.228587 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260806 16:06:56.432431 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:56.432007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:48971)
0806 16:06:56.432274 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:56.432332 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
I20260806 16:06:57.069013 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:57.068593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:44421)
0806 16:06:57.068855 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:57.068914 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260806 16:06:57.229919 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:57.229451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:44407)
0806 16:06:57.229758 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:57.229817 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
I20260806 16:06:57.433667 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:57.433177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:34877)
0806 16:06:57.433497 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:57.433559 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260806 16:06:58.070331 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:58.069854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:43315)
0806 16:06:58.070158 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:58.070224 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260806 16:06:58.231421 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:58.230960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:42641)
0806 16:06:58.231245 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:58.231312 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260806 16:06:58.318315 3401 raft_consensus.cc:482] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:06:58.318547 3401 raft_consensus.cc:504] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106164208bf46c5b673c9fdca76e2ae" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43725 } } peers { permanent_uuid: "5315bfa3b8414931999644babe6e6a48" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36051 } } peers { permanent_uuid: "1679b864b5d44183b9233c55f5913ef4" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43851 } }
I20260806 16:06:58.319149 3401 leader_election.cc:288] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725), 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051)
W20260806 16:06:58.320087 2806 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c106164208bf46c5b673c9fdca76e2ae (127.13.148.2:43725): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
W20260806 16:06:58.320204 2803 leader_election.cc:341] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5315bfa3b8414931999644babe6e6a48 (127.13.148.3:36051): Not found: Tablet not found: f3772ea70c944e93a072c63dc24e4ce3
I20260806 16:06:58.320262 2803 leader_election.cc:302] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1679b864b5d44183b9233c55f5913ef4; no voters: 5315bfa3b8414931999644babe6e6a48, c106164208bf46c5b673c9fdca76e2ae
I20260806 16:06:58.320451 3401 raft_consensus.cc:2726] T f3772ea70c944e93a072c63dc24e4ce3 P 1679b864b5d44183b9233c55f5913ef4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:06:58.435333 3402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:58.434811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:44527)
0806 16:06:58.435154 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:58.435226 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260806 16:06:59.071653 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:59.071149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:59677)
0806 16:06:59.071455 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:59.071518 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":104,"threads_started":1}
I20260806 16:06:59.232743 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:59.232289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:50611)
0806 16:06:59.232576 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:59.232638 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260806 16:06:59.436594 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:06:59.436164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:49013)
0806 16:06:59.436430 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 16:06:59.436491 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260806 16:07:00.073313 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:00.072845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.1:57831)
0806 16:07:00.073140 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:00.073201 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":77,"threads_started":1}
I20260806 16:07:00.234220 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:00.233732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.2:55425)
0806 16:07:00.234044 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:00.234111 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260806 16:07:00.423836 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 2738
I20260806 16:07:00.436328 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 2962
I20260806 16:07:00.441071 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:00.440421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39593 (local address 127.13.148.3:53363)
0806 16:07:00.440910 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:00.440967 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":85,"threads_started":1}
I20260806 16:07:00.448343 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 3150
2026-08-06T16:07:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19638 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:07:00 dist-test-slave-7kzw krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:07:00 dist-test-slave-7kzw krb5kdc[3414](info): set up 2 sockets
Aug 06 16:07:00 dist-test-slave-7kzw krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260806 16:07:01.710328 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:07:01 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:07:01.847678 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35479
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:07:01.969457 3430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:01.969802 3430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:01.969884 3430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:01.973969 3430 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 16:07:01.974050 3430 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 16:07:01.974079 3430 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 16:07:01.974113 3430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:07:01.974151 3430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:01.974190 3430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:01.974231 3430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:07:01.974268 3430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:07:01.979789 3430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:01.980926 3430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:01.982036 3430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:01.991314 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:01.991379 3430 server_base.cc:1034] running on GCE node
W20260806 16:07:01.991318 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:01.991321 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:01.992260 3430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:01.993407 3430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:01.994633 3430 hybrid_clock.cc:648] HybridClock initialized: now 1786032421994606 us; error 33 us; skew 500 ppm
Aug 06 16:07:01 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032421, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:01.997632 3430 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:07:01.998589 3430 webserver.cc:460] Webserver started at http://127.13.148.62:38247/ using document root <none> and password file <none>
I20260806 16:07:01.999191 3430 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:01.999293 3430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:01.999534 3430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:07:02.001648 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "df0982a3339e4d30ae404cdf5ed3d524"
format_stamp: "Formatted at 2026-08-06 16:07:01 on dist-test-slave-7kzw"
server_key: "3e431af03247d91faddd965283ebbc47"
server_key_iv: "d70acbd4e5c5329959728e840e1a5221"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.002262 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "df0982a3339e4d30ae404cdf5ed3d524"
format_stamp: "Formatted at 2026-08-06 16:07:01 on dist-test-slave-7kzw"
server_key: "3e431af03247d91faddd965283ebbc47"
server_key_iv: "d70acbd4e5c5329959728e840e1a5221"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.006230 3430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 16:07:02.008930 3451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.010177 3430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:07:02.010331 3430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "df0982a3339e4d30ae404cdf5ed3d524"
format_stamp: "Formatted at 2026-08-06 16:07:01 on dist-test-slave-7kzw"
server_key: "3e431af03247d91faddd965283ebbc47"
server_key_iv: "d70acbd4e5c5329959728e840e1a5221"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.010455 3430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:02.035863 3430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:02.039539 3430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:02.039768 3430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:02.047307 3430 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35479
I20260806 16:07:02.047657 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35479 every 8 connection(s)
I20260806 16:07:02.048333 3430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:07:02.051332 3523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:07:02.056173 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 3430
I20260806 16:07:02.056361 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:07:02.056640 13904 external_mini_cluster.cc:1428] Setting key 146930da186df33587f7bc78a9c1966d
I20260806 16:07:02.057904 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Bootstrap starting.
I20260806 16:07:02.060539 3523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Neither blocks nor log segments found. Creating new log.
I20260806 16:07:02.061400 3523 log.cc:824] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:02.063575 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: No bootstrap required, opened a new log
Aug 06 16:07:02 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:07:02.068740 3523 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:02.069021 3523 raft_consensus.cc:374] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:07:02.069090 3523 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0982a3339e4d30ae404cdf5ed3d524, State: Initialized, Role: FOLLOWER
I20260806 16:07:02.069624 3523 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:02.069787 3523 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:07:02.069880 3523 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:07:02.070001 3523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:07:02.071260 3523 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:02.071668 3523 leader_election.cc:302] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0982a3339e4d30ae404cdf5ed3d524; no voters:
I20260806 16:07:02.072018 3523 leader_election.cc:288] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:07:02.072142 3531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:07:02.072361 3531 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 1 LEADER]: Becoming Leader. State: Replica: df0982a3339e4d30ae404cdf5ed3d524, State: Running, Role: LEADER
I20260806 16:07:02.072785 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.058189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.13.148.62:35479)
0806 16:07:02.058750 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.058766 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.058806 (+ 40us) server_negotiation.cc:407] Connection header received
0806 16:07:02.059482 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.059500 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.059846 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.060191 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:02.061054 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.062081 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.063019 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.063496 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.063673 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.066725 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:02.066731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:02.066745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:02.066775 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:02.068893 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.069465 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.069471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.069476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.069618 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.069949 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.069953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.069954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.070456 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:02.070762 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.071249 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.071618 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":165,"threads_started":1}
I20260806 16:07:02.072759 3531 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:02.073511 3523 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:07:02.074059 3535 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0982a3339e4d30ae404cdf5ed3d524. Latest consensus state: current_term: 1 leader_uuid: "df0982a3339e4d30ae404cdf5ed3d524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } } }
I20260806 16:07:02.074184 3535 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: This master's current role is: LEADER
I20260806 16:07:02.074913 3534 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0982a3339e4d30ae404cdf5ed3d524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } } }
I20260806 16:07:02.075006 3534 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: This master's current role is: LEADER
I20260806 16:07:02.075325 3540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:07:02.078241 3540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:07:02.084517 3540 catalog_manager.cc:1347] Generated new cluster ID: 875e9463cd0e420b879dd25ae80bbd38
I20260806 16:07:02.084693 3540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:07:02.106740 3540 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:07:02.107822 3540 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:07:02.112528 3540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Generated new TSK 0
I20260806 16:07:02.113283 3540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:07:02.154352 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35479
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:07:02.273751 3564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:02.274122 3564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:02.274173 3564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:02.278606 3564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:02.278676 3564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:02.278735 3564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:07:02.284214 3564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:02.285272 3564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:02.286271 3564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:02.295984 3571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:02.296195 3564 server_base.cc:1034] running on GCE node
W20260806 16:07:02.296018 3572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:02.295969 3575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:02.296831 3564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:02.297427 3564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:02.298535 3564 hybrid_clock.cc:648] HybridClock initialized: now 1786032422298520 us; error 25 us; skew 500 ppm
Aug 06 16:07:02 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032422, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:02.301316 3564 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:07:02.302146 3564 webserver.cc:460] Webserver started at http://127.13.148.1:41759/ using document root <none> and password file <none>
I20260806 16:07:02.302798 3564 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:02.302894 3564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:02.303128 3564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:07:02.305099 3564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "68639b9520c94d70b832d7c52918c0dc"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "d3addd862c35bad83742764be7333a53"
server_key_iv: "30eebee35c9fc1f42ee0422323474ccf"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.305677 3564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "68639b9520c94d70b832d7c52918c0dc"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "d3addd862c35bad83742764be7333a53"
server_key_iv: "30eebee35c9fc1f42ee0422323474ccf"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.309496 3564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:07:02.311966 3582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.313181 3564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:07:02.313335 3564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "68639b9520c94d70b832d7c52918c0dc"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "d3addd862c35bad83742764be7333a53"
server_key_iv: "30eebee35c9fc1f42ee0422323474ccf"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.313450 3564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:02.325600 3564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:02.328644 3564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:02.328843 3564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:02.329433 3564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:02.330248 3564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:07:02.330322 3564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.330389 3564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:07:02.330441 3564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.340095 3564 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:37149
I20260806 16:07:02.340132 3739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:37149 every 8 connection(s)
I20260806 16:07:02.341037 3564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:07:02.342226 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 3564
I20260806 16:07:02.342372 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:07:02.342778 13904 external_mini_cluster.cc:1428] Setting key f987f7ac061f90f21d685c61cd191079
Aug 06 16:07:02 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032422, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:07:02.357961 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.343348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:50727 (local address 127.13.148.62:35479)
0806 16:07:02.343526 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.343532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.344619 (+ 1087us) server_negotiation.cc:407] Connection header received
0806 16:07:02.346765 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.346768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.346869 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.346986 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:02.348331 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.348927 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.350196 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.350509 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.350660 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.353775 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:02.353781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:02.353787 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:02.353819 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:02.355482 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.356028 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.356031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.356032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.356116 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.356648 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.356651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.356652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.356839 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:02.356928 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.357572 (+ 644us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.357717 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:07:02.359231 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.343835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:50727)
0806 16:07:02.344467 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:02.344512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:02.346516 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:02.347214 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:02.347222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:02.347634 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:07:02.348160 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.348175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.349074 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.349078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.350041 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.350052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.350696 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.350701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.350908 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.351635 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:07:02.351654 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:07:02.353535 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:07:02.355619 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:02.355624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:02.355639 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:02.355928 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:02.356258 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:02.356338 (+ 80us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:02.356343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:02.356550 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:02.356959 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:07:02.356973 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:07:02.357296 (+ 323us) client_negotiation.cc:769] Sending connection context
0806 16:07:02.357523 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 16:07:02.357956 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":190,"threads_started":1}
I20260806 16:07:02.360816 3740 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35479
I20260806 16:07:02.361145 3740 heartbeater.cc:464] Registering TS with master...
I20260806 16:07:02.361840 3740 heartbeater.cc:511] Master 127.13.148.62:35479 requested a full tablet report, sending...
I20260806 16:07:02.363937 3477 ts_manager.cc:194] Registered new tserver with Master: 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149)
I20260806 16:07:02.365561 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:50727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:07:02.378865 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35479
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:07:02.509822 3746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:02.510253 3746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:02.510350 3746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:02.519711 3746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:02.519815 3746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:02.519948 3746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:07:02.531430 3746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:02.533159 3746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:02.534718 3746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:02.548110 3759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:02.548156 3758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:02.548146 3761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:02.548600 3746 server_base.cc:1034] running on GCE node
I20260806 16:07:02.549023 3746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:02.549692 3746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:02.550861 3746 hybrid_clock.cc:648] HybridClock initialized: now 1786032422550846 us; error 26 us; skew 500 ppm
Aug 06 16:07:02 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032422, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:02.553961 3746 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:07:02.555063 3746 webserver.cc:460] Webserver started at http://127.13.148.2:39933/ using document root <none> and password file <none>
I20260806 16:07:02.555701 3746 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:02.555804 3746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:02.556048 3746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:07:02.558233 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "816b527aff5e4c91b166164e715fdafe"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "be7606e789e122f9f62f6d3647f31e68"
server_key_iv: "9be8c8fc22e2aee082b090d3db5c793d"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.558923 3746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "816b527aff5e4c91b166164e715fdafe"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "be7606e789e122f9f62f6d3647f31e68"
server_key_iv: "9be8c8fc22e2aee082b090d3db5c793d"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.563067 3746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:07:02.565742 3771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.567157 3746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:07:02.567329 3746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "816b527aff5e4c91b166164e715fdafe"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "be7606e789e122f9f62f6d3647f31e68"
server_key_iv: "9be8c8fc22e2aee082b090d3db5c793d"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.567456 3746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:02.597597 3746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:02.601158 3746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:02.601373 3746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:02.602001 3746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:02.602937 3746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:07:02.603012 3746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.603087 3746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:07:02.603139 3746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.613608 3746 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:41565
I20260806 16:07:02.613658 3921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:41565 every 8 connection(s)
I20260806 16:07:02.614751 3746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:07:02.617905 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 3746
I20260806 16:07:02.618027 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:07:02.618294 13904 external_mini_cluster.cc:1428] Setting key 945c2ccda3cb08d3dc05471c6dd93442
Aug 06 16:07:02 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032422, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:07:02.633704 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.616711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33373 (local address 127.13.148.62:35479)
0806 16:07:02.616878 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.616884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.617898 (+ 1014us) server_negotiation.cc:407] Connection header received
0806 16:07:02.619991 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.619994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.620047 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.620156 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:02.621658 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.622413 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.625919 (+ 3506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.626259 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.626368 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.629288 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:02.629295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:02.629301 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:02.629349 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:02.631195 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.631720 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.631726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.631731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.631835 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.632266 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.632273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.632278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.632562 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:02.632695 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.633274 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.633429 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:07:02.634423 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.617087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:33373)
0806 16:07:02.617671 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:02.617791 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:02.619742 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:02.620342 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:02.620351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:02.620825 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:07:02.621470 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.621488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.624898 (+ 3410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.624901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.625788 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.625797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.626452 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.626455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.626692 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.627364 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:07:02.627382 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:07:02.629101 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:07:02.631320 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:02.631325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:02.631338 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:02.631610 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:02.631973 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:02.631976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:02.631977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:02.632145 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:02.632722 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:07:02.632728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:07:02.632986 (+ 258us) client_negotiation.cc:769] Sending connection context
0806 16:07:02.633182 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 16:07:02.633468 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":120,"threads_started":1}
I20260806 16:07:02.635818 3922 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35479
I20260806 16:07:02.636107 3922 heartbeater.cc:464] Registering TS with master...
I20260806 16:07:02.636790 3922 heartbeater.cc:511] Master 127.13.148.62:35479 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:07:02.638075 3477 ts_manager.cc:194] Registered new tserver with Master: 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565)
I20260806 16:07:02.638871 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:33373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:07:02.658165 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35479
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:07:02.772367 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:02.772677 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:02.772753 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:02.777066 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:02.777158 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:02.777252 3929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:07:02.782684 3929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:02.783782 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:02.784830 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:02.794737 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:02.794744 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:02.794973 3929 server_base.cc:1034] running on GCE node
W20260806 16:07:02.794894 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:02.795498 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:02.796072 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:02.797247 3929 hybrid_clock.cc:648] HybridClock initialized: now 1786032422797212 us; error 47 us; skew 500 ppm
Aug 06 16:07:02 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032422, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:02.800302 3929 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:07:02.801200 3929 webserver.cc:460] Webserver started at http://127.13.148.3:41653/ using document root <none> and password file <none>
I20260806 16:07:02.801777 3929 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:02.801873 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:02.802129 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:07:02.804258 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "67de039cef5f42fead896ffb9c4432f2"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "b629b417006f2e506080555993fe79f0"
server_key_iv: "d4dfbecc957a51726f9c1d160da74fed"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.804920 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "67de039cef5f42fead896ffb9c4432f2"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "b629b417006f2e506080555993fe79f0"
server_key_iv: "d4dfbecc957a51726f9c1d160da74fed"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.808928 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:07:02.811812 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.813190 3929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:07:02.813360 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "67de039cef5f42fead896ffb9c4432f2"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "b629b417006f2e506080555993fe79f0"
server_key_iv: "d4dfbecc957a51726f9c1d160da74fed"
server_key_version: "encryptionkey@0"
I20260806 16:07:02.813475 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:02.828696 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:02.832075 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:02.832293 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:02.832937 3929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:02.833814 3929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:07:02.833890 3929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.833966 3929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:07:02.834012 3929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:02.843641 3929 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:34683
I20260806 16:07:02.843662 4096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:34683 every 8 connection(s)
I20260806 16:07:02.844547 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:07:02.845659 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 3929
I20260806 16:07:02.845790 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:07:02.846072 13904 external_mini_cluster.cc:1428] Setting key 9c039e3d2a45047a4aaa7f73b9d453da
Aug 06 16:07:02 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032422, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:07:02.858510 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.846565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:37881 (local address 127.13.148.62:35479)
0806 16:07:02.846730 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.846742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.847688 (+ 946us) server_negotiation.cc:407] Connection header received
0806 16:07:02.849204 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.849206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.849255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.849325 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:02.850445 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.851029 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.852187 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.852401 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.852477 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.855044 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:02.855047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:02.855049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:02.855073 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:02.856568 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.857057 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.857060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.857061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.857139 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.857440 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.857443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.857444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.857652 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:02.857732 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.858215 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.858327 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 16:07:02.859360 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.847001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:37881)
0806 16:07:02.847530 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:02.847598 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:02.849004 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:02.849456 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:02.849462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:02.849833 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:07:02.850314 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.850326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.851156 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.851159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.852072 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.852083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.852530 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.852533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.852709 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.853352 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:07:02.853369 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:07:02.854882 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:07:02.856702 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:02.856706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:02.856717 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:02.856970 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:02.857218 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:02.857221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:02.857222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:02.857361 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:02.857763 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:07:02.857768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:07:02.857994 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 16:07:02.858149 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 16:07:02.858400 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":120,"threads_started":1}
I20260806 16:07:02.860630 4097 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35479
I20260806 16:07:02.860872 4097 heartbeater.cc:464] Registering TS with master...
I20260806 16:07:02.861366 4097 heartbeater.cc:511] Master 127.13.148.62:35479 requested a full tablet report, sending...
I20260806 16:07:02.862424 3475 ts_manager.cc:194] Registered new tserver with Master: 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
I20260806 16:07:02.863025 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:37881
I20260806 16:07:02.872592 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:07:02.883325 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.875026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.13.148.62:35479)
0806 16:07:02.875250 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.875257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.875492 (+ 235us) server_negotiation.cc:407] Connection header received
0806 16:07:02.875659 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.875664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.875732 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.875842 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:02.876647 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.877198 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.878136 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.878364 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.878584 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.880121 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:02.880124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:02.880126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:02.880170 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:02.881559 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.882003 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.882005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.882007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.882088 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:02.882399 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:02.882401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:02.882403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:02.882717 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:02.882828 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.883088 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.883197 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 16:07:02.886970 3475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:07:02.890295 3475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:07:02.906520 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.900184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.13.148.3:34683)
0806 16:07:02.901377 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.901388 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.901407 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:07:02.901470 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.901474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.901613 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.902025 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:02.902177 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.903021 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.904973 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.905826 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.905944 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.906088 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.906274 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.906348 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":115,"threads_started":1}
I20260806 16:07:02.906584 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.899816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34683 (local address 127.0.0.1:51876)
0806 16:07:02.900719 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:02.900775 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:02.900916 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:02.901775 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:02.901778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:02.901805 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:02.902085 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.902095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.903144 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.903150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.904812 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.904823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.905958 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.905962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.906069 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.906113 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 16:07:02.906254 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 16:07:02.906394 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":251,"spinlock_wait_cycles":6656,"thread_start_us":61,"threads_started":1}
I20260806 16:07:02.907037 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.900504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41565 (local address 127.0.0.1:33106)
0806 16:07:02.901160 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:02.901172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:02.901258 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:02.901765 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:02.901768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:02.901782 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:02.902031 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.902039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.903452 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.903455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.904803 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.904815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.906576 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.906582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.906725 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.906757 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 16:07:02.906834 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 16:07:02.906900 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":231,"spinlock_wait_cycles":8576,"thread_start_us":90,"threads_started":1}
I20260806 16:07:02.907136 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.900680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.13.148.2:41565)
0806 16:07:02.901146 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.901162 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.901334 (+ 172us) server_negotiation.cc:407] Connection header received
0806 16:07:02.901422 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.901427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.901607 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.901772 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:02.902180 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.903294 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.904942 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.905826 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.905944 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.906100 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.906889 (+ 789us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.906983 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":147,"threads_started":1}
I20260806 16:07:02.907357 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.899920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.13.148.1:37149)
0806 16:07:02.900554 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.900562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.901068 (+ 506us) server_negotiation.cc:407] Connection header received
0806 16:07:02.901196 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.901205 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.901438 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.901597 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:02.901897 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.903158 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.905578 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.906284 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.906381 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.906757 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.907106 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.907209 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":122,"threads_started":1}
I20260806 16:07:02.907612 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.899806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37149 (local address 127.0.0.1:55218)
0806 16:07:02.900719 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:02.900775 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:02.900916 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:02.901595 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:02.901601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:02.901623 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:02.901762 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.901769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.903278 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.903281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.905452 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.905461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.906553 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.906557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.906661 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.906798 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 16:07:02.906897 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:07:02.906971 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":153,"spinlock_wait_cycles":7168,"thread_start_us":127,"threads_started":1}
I20260806 16:07:02.910334 4011 tablet_service.cc:1467] Processing CreateTablet for tablet 569267b26eef4316b170702fdbfdadf0 (DEFAULT_TABLE table=test-table [id=cb60bb2c00f548089504faef2c6ac321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:07:02.910343 3648 tablet_service.cc:1467] Processing CreateTablet for tablet 569267b26eef4316b170702fdbfdadf0 (DEFAULT_TABLE table=test-table [id=cb60bb2c00f548089504faef2c6ac321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:07:02.910347 3832 tablet_service.cc:1467] Processing CreateTablet for tablet 569267b26eef4316b170702fdbfdadf0 (DEFAULT_TABLE table=test-table [id=cb60bb2c00f548089504faef2c6ac321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:07:02.911441 3832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569267b26eef4316b170702fdbfdadf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:07:02.911441 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569267b26eef4316b170702fdbfdadf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:07:02.911545 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569267b26eef4316b170702fdbfdadf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:07:02.918649 4127 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: Bootstrap starting.
I20260806 16:07:02.919790 4128 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Bootstrap starting.
I20260806 16:07:02.921152 4127 tablet_bootstrap.cc:654] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: Neither blocks nor log segments found. Creating new log.
I20260806 16:07:02.921581 4128 tablet_bootstrap.cc:654] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Neither blocks nor log segments found. Creating new log.
I20260806 16:07:02.922303 4128 log.cc:824] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:02.922302 4127 log.cc:824] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:02.922780 4129 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: Bootstrap starting.
I20260806 16:07:02.924590 4128 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: No bootstrap required, opened a new log
I20260806 16:07:02.924600 4127 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: No bootstrap required, opened a new log
I20260806 16:07:02.924796 4128 ts_tablet_manager.cc:1397] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 16:07:02.924796 4127 ts_tablet_manager.cc:1397] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:07:02.925143 4129 tablet_bootstrap.cc:654] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: Neither blocks nor log segments found. Creating new log.
I20260806 16:07:02.925825 4129 log.cc:824] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:02.927577 4129 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: No bootstrap required, opened a new log
I20260806 16:07:02.927783 4129 ts_tablet_manager.cc:1397] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:07:02.929219 4127 raft_consensus.cc:348] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:02.929550 4127 raft_consensus.cc:374] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:07:02.929636 4127 raft_consensus.cc:729] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67de039cef5f42fead896ffb9c4432f2, State: Initialized, Role: FOLLOWER
I20260806 16:07:02.929586 4128 raft_consensus.cc:348] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:02.929874 4128 raft_consensus.cc:374] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:07:02.929942 4128 raft_consensus.cc:729] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68639b9520c94d70b832d7c52918c0dc, State: Initialized, Role: FOLLOWER
I20260806 16:07:02.930220 4127 consensus_queue.cc:260] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:02.930369 4128 consensus_queue.cc:260] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:02.931006 4097 heartbeater.cc:503] Master 127.13.148.62:35479 was elected leader, sending a full tablet report...
I20260806 16:07:02.930989 4129 raft_consensus.cc:348] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:02.931269 4129 raft_consensus.cc:374] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:07:02.931339 4129 raft_consensus.cc:729] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816b527aff5e4c91b166164e715fdafe, State: Initialized, Role: FOLLOWER
I20260806 16:07:02.931416 4128 ts_tablet_manager.cc:1428] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 16:07:02.931790 3740 heartbeater.cc:503] Master 127.13.148.62:35479 was elected leader, sending a full tablet report...
I20260806 16:07:02.931747 4129 consensus_queue.cc:260] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:02.932603 4127 ts_tablet_manager.cc:1428] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 16:07:02.932847 3922 heartbeater.cc:503] Master 127.13.148.62:35479 was elected leader, sending a full tablet report...
I20260806 16:07:02.932896 4129 ts_tablet_manager.cc:1428] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 16:07:02.993288 4136 raft_consensus.cc:482] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:07:02.993526 4136 raft_consensus.cc:504] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:02.994745 4136 leader_election.cc:288] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149), 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
I20260806 16:07:02.999934 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.995155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:49189 (local address 127.13.148.3:34683)
0806 16:07:02.995322 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.995327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.995346 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:07:02.995405 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.995414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.995455 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.995522 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:02.995854 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.996644 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.998569 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.999195 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.999337 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.999381 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.999733 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.999807 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:07:02.999958 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.995164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:54853 (local address 127.13.148.1:37149)
0806 16:07:02.995300 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:02.995305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:02.995579 (+ 274us) server_negotiation.cc:407] Connection header received
0806 16:07:02.995677 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:02.995680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:02.995720 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:02.995846 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:02.996072 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.996792 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.998572 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.999196 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.999327 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.999381 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:02.999732 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 16:07:02.999854 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:07:02.999943 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.994998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34683 (local address 127.13.148.2:49189)
0806 16:07:02.995174 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:02.995192 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:02.995297 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:02.995568 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:02.995571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:02.995595 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:02.995741 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.995746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.996784 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.996788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.998424 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.998437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.999344 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.999348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.999574 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.999604 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 16:07:02.999693 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 16:07:02.999778 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":138,"spinlock_wait_cycles":3840}
I20260806 16:07:02.999951 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:02.994998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37149 (local address 127.13.148.2:54853)
0806 16:07:02.995451 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:02.995468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:02.995587 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:02.995852 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:02.995857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:02.995876 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:02.995967 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.995971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.996909 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.996913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.998424 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:02.998437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:02.999344 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:02.999348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:02.999604 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:02.999623 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 16:07:02.999693 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 16:07:02.999778 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":299,"thread_start_us":132,"threads_started":1}
I20260806 16:07:03.000742 4038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569267b26eef4316b170702fdbfdadf0" candidate_uuid: "816b527aff5e4c91b166164e715fdafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67de039cef5f42fead896ffb9c4432f2" is_pre_election: true
I20260806 16:07:03.000957 3674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569267b26eef4316b170702fdbfdadf0" candidate_uuid: "816b527aff5e4c91b166164e715fdafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68639b9520c94d70b832d7c52918c0dc" is_pre_election: true
I20260806 16:07:03.001086 4038 raft_consensus.cc:2454] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816b527aff5e4c91b166164e715fdafe in term 0.
I20260806 16:07:03.001291 3674 raft_consensus.cc:2454] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816b527aff5e4c91b166164e715fdafe in term 0.
I20260806 16:07:03.001626 3777 leader_election.cc:302] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe; no voters:
I20260806 16:07:03.001955 4136 raft_consensus.cc:2781] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:07:03.002044 4136 raft_consensus.cc:482] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:07:03.002090 4136 raft_consensus.cc:3037] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:07:03.003268 4136 raft_consensus.cc:504] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:03.003666 4136 leader_election.cc:288] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [CANDIDATE]: Term 1 election: Requested vote from peers 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149), 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
I20260806 16:07:03.004030 4038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569267b26eef4316b170702fdbfdadf0" candidate_uuid: "816b527aff5e4c91b166164e715fdafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67de039cef5f42fead896ffb9c4432f2"
I20260806 16:07:03.004146 4038 raft_consensus.cc:3037] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:07:03.004097 3674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569267b26eef4316b170702fdbfdadf0" candidate_uuid: "816b527aff5e4c91b166164e715fdafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68639b9520c94d70b832d7c52918c0dc"
I20260806 16:07:03.004254 3674 raft_consensus.cc:3037] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:07:03.005167 4038 raft_consensus.cc:2454] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816b527aff5e4c91b166164e715fdafe in term 1.
I20260806 16:07:03.005401 3674 raft_consensus.cc:2454] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816b527aff5e4c91b166164e715fdafe in term 1.
I20260806 16:07:03.005465 3777 leader_election.cc:302] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe; no voters:
I20260806 16:07:03.005693 4136 raft_consensus.cc:2781] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:07:03.005962 4136 raft_consensus.cc:686] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 1 LEADER]: Becoming Leader. State: Replica: 816b527aff5e4c91b166164e715fdafe, State: Running, Role: LEADER
I20260806 16:07:03.006317 4136 consensus_queue.cc:237] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:03.009867 3477 catalog_manager.cc:5495] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe reported cstate change: term changed from 0 to 1, leader changed from <none> to 816b527aff5e4c91b166164e715fdafe (127.13.148.2). New cstate: current_term: 1 leader_uuid: "816b527aff5e4c91b166164e715fdafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } health_report { overall_health: HEALTHY } } }
I20260806 16:07:03.035709 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:03.032001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.13.148.2:41565)
0806 16:07:03.032145 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:03.032149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:03.032162 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:07:03.032244 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:03.032246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:03.032307 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:03.032405 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:07:03.032728 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:03.033254 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:03.034167 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:03.034378 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:03.034515 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:03.034845 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:07:03.035314 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:07:03.035431 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:03.035502 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 16:07:03.035584 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:07:03.041963 4038 raft_consensus.cc:1261] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 1 FOLLOWER]: Refusing update from remote peer 816b527aff5e4c91b166164e715fdafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:07:03.042035 3674 raft_consensus.cc:1261] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Refusing update from remote peer 816b527aff5e4c91b166164e715fdafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:07:03.042722 4136 consensus_queue.cc:1035] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:07:03.042976 4138 consensus_queue.cc:1035] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:07:03.050688 4144 mvcc.cc:205] Tried to move back new op lower bound from 7315588804772376576 to 7315588804637159424. Current Snapshot: MvccSnapshot[applied={T|T < 7315588804772376576}]
I20260806 16:07:03.053634 4142 mvcc.cc:205] Tried to move back new op lower bound from 7315588804772376576 to 7315588804637159424. Current Snapshot: MvccSnapshot[applied={T|T < 7315588804772376576}]
I20260806 16:07:03.057080 4143 mvcc.cc:205] Tried to move back new op lower bound from 7315588804772376576 to 7315588804637159424. Current Snapshot: MvccSnapshot[applied={T|T < 7315588804772376576}]
I20260806 16:07:03.062582 3475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:07:03.062790 3475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:07:03.065459 3475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Sending DeleteTablet for 3 replicas of tablet 569267b26eef4316b170702fdbfdadf0
I20260806 16:07:03.066325 3648 tablet_service.cc:1514] Processing DeleteTablet for tablet 569267b26eef4316b170702fdbfdadf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:07:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:55218
I20260806 16:07:03.066382 4011 tablet_service.cc:1514] Processing DeleteTablet for tablet 569267b26eef4316b170702fdbfdadf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:07:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51876
I20260806 16:07:03.066567 3832 tablet_service.cc:1514] Processing DeleteTablet for tablet 569267b26eef4316b170702fdbfdadf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:07:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:33106
I20260806 16:07:03.066713 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 3564
I20260806 16:07:03.077713 4154 tablet_replica.cc:330] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: stopping tablet replica
I20260806 16:07:03.078266 4154 raft_consensus.cc:2229] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:07:03.078722 4154 raft_consensus.cc:2258] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:07:03.079209 4156 tablet_replica.cc:330] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: stopping tablet replica
I20260806 16:07:03.079548 4156 raft_consensus.cc:2229] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:07:03.080070 4156 raft_consensus.cc:2258] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:07:03.081966 4156 ts_tablet_manager.cc:1905] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:07:03.084664 4154 ts_tablet_manager.cc:1905] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:07:03.085700 4156 ts_tablet_manager.cc:1918] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:07:03.085829 4156 log.cc:1196] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/569267b26eef4316b170702fdbfdadf0
I20260806 16:07:03.086231 4156 ts_tablet_manager.cc:1939] T 569267b26eef4316b170702fdbfdadf0 P 816b527aff5e4c91b166164e715fdafe: Deleting consensus metadata
W20260806 16:07:03.086804 3455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:37149 (error 108)
I20260806 16:07:03.087867 3457 catalog_manager.cc:4841] TS 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565): tablet 569267b26eef4316b170702fdbfdadf0 (table test-table [id=cb60bb2c00f548089504faef2c6ac321]) successfully deleted
I20260806 16:07:03.088078 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:03.087662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37149 (local address 127.0.0.1:55228)
0806 16:07:03.087835 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:03.087944 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:37149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 16:07:03.088239 3455 catalog_manager.cc:4584] TS 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 569267b26eef4316b170702fdbfdadf0: Network error: Client connection negotiation failed: client connection to 127.13.148.1:37149: connect: Connection refused (error 111)
I20260806 16:07:03.088711 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 3746
I20260806 16:07:03.093360 4154 ts_tablet_manager.cc:1918] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:07:03.093478 4154 log.cc:1196] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/569267b26eef4316b170702fdbfdadf0
I20260806 16:07:03.093883 4154 ts_tablet_manager.cc:1939] T 569267b26eef4316b170702fdbfdadf0 P 67de039cef5f42fead896ffb9c4432f2: Deleting consensus metadata
I20260806 16:07:03.095453 3457 catalog_manager.cc:4841] TS 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683): tablet 569267b26eef4316b170702fdbfdadf0 (table test-table [id=cb60bb2c00f548089504faef2c6ac321]) successfully deleted
I20260806 16:07:03.103112 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 3929
I20260806 16:07:03.123974 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 3430
I20260806 16:07:23.140892 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35479
--webserver_interface=127.13.148.62
--webserver_port=38247
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:07:23.274775 4158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:23.275202 4158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:23.275290 4158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:23.280984 4158 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 16:07:23.281057 4158 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 16:07:23.281090 4158 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 16:07:23.281114 4158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:07:23.281138 4158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:23.281188 4158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:23.281217 4158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:07:23.281252 4158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:07:23.286862 4158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:23.288327 4158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:23.289407 4158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:23.304538 4167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:23.304562 4165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:23.304682 4158 server_base.cc:1034] running on GCE node
W20260806 16:07:23.304562 4164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:23.305665 4158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:23.307020 4158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:23.308244 4158 hybrid_clock.cc:648] HybridClock initialized: now 1786032443308235 us; error 35 us; skew 500 ppm
Aug 06 16:07:23 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032443, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:23.311792 4158 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:07:23.312826 4158 webserver.cc:460] Webserver started at http://127.13.148.62:38247/ using document root <none> and password file <none>
I20260806 16:07:23.313506 4158 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:23.313635 4158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:23.318094 4158 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:07:23.320762 4177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:23.322263 4158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 16:07:23.322445 4158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "df0982a3339e4d30ae404cdf5ed3d524"
format_stamp: "Formatted at 2026-08-06 16:07:01 on dist-test-slave-7kzw"
server_key: "3e431af03247d91faddd965283ebbc47"
server_key_iv: "d70acbd4e5c5329959728e840e1a5221"
server_key_version: "encryptionkey@0"
I20260806 16:07:23.323199 4158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:23.346819 4158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:23.350286 4158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:23.350550 4158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:23.358840 4158 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35479
I20260806 16:07:23.359167 4244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35479 every 8 connection(s)
I20260806 16:07:23.360071 4158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:07:23.361142 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 4158
I20260806 16:07:23.361649 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:37149
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:35479
--webserver_port=41759
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:07:23.364955 4245 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:07:23.366793 4245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Bootstrap starting.
I20260806 16:07:23.384642 4245 log.cc:824] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:23.393698 4245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:07:23.394153 4245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Bootstrap complete.
I20260806 16:07:23.398406 4245 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:23.398937 4245 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0982a3339e4d30ae404cdf5ed3d524, State: Initialized, Role: FOLLOWER
I20260806 16:07:23.399506 4245 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:23.399627 4245 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:07:23.399717 4245 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:07:23.399845 4245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 1 FOLLOWER]: Advancing to term 2
I20260806 16:07:23.401746 4245 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:23.402091 4245 leader_election.cc:302] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0982a3339e4d30ae404cdf5ed3d524; no voters:
I20260806 16:07:23.402442 4245 leader_election.cc:288] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 16:07:23.402556 4251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 16:07:23.402782 4251 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 2 LEADER]: Becoming Leader. State: Replica: df0982a3339e4d30ae404cdf5ed3d524, State: Running, Role: LEADER
I20260806 16:07:23.403075 4251 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:23.403699 4245 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:07:23.404068 4254 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0982a3339e4d30ae404cdf5ed3d524. Latest consensus state: current_term: 2 leader_uuid: "df0982a3339e4d30ae404cdf5ed3d524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } } }
I20260806 16:07:23.404203 4254 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: This master's current role is: LEADER
I20260806 16:07:23.404490 4252 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "df0982a3339e4d30ae404cdf5ed3d524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } } }
I20260806 16:07:23.404619 4252 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: This master's current role is: LEADER
I20260806 16:07:23.404855 4258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:07:23.410882 4258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:07:23.411762 4258 catalog_manager.cc:1259] Loaded cluster ID: 875e9463cd0e420b879dd25ae80bbd38
I20260806 16:07:23.411836 4258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:07:23.414853 4258 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:07:23.415642 4258 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Loaded TSK: 0
I20260806 16:07:23.433629 4258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Generated new TSK 1
I20260806 16:07:23.433918 4258 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:07:23.494879 4248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:23.495254 4248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:23.495350 4248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:23.499363 4248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:23.499421 4248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:23.499524 4248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:07:23.505286 4248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:37149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=41759
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:23.506417 4248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:23.507534 4248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:23.518819 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:23.518819 4286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:23.519045 4248 server_base.cc:1034] running on GCE node
W20260806 16:07:23.518850 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:23.519718 4248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:23.520403 4248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:23.521600 4248 hybrid_clock.cc:648] HybridClock initialized: now 1786032443521571 us; error 43 us; skew 500 ppm
Aug 06 16:07:23 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032443, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:23.524835 4248 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:07:23.525877 4248 webserver.cc:460] Webserver started at http://127.13.148.1:41759/ using document root <none> and password file <none>
I20260806 16:07:23.526539 4248 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:23.526666 4248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:23.530961 4248 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:07:23.533289 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:23.534785 4248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:07:23.534945 4248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "68639b9520c94d70b832d7c52918c0dc"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "d3addd862c35bad83742764be7333a53"
server_key_iv: "30eebee35c9fc1f42ee0422323474ccf"
server_key_version: "encryptionkey@0"
I20260806 16:07:23.535528 4248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:23.550174 4248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:23.553692 4248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:23.553916 4248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:23.554600 4248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:23.555785 4307 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 16:07:23.560141 4248 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 16:07:23.560242 4248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 16:07:23.560333 4248 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 16:07:23.562026 4248 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 16:07:23.562099 4248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 16:07:23.562533 4307 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Bootstrap starting.
I20260806 16:07:23.573314 4248 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:37149
I20260806 16:07:23.573351 4464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:37149 every 8 connection(s)
I20260806 16:07:23.574393 4248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:07:23.580461 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 4248
I20260806 16:07:23.580987 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:41565
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:35479
--webserver_port=39933
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:07:23.581589 4307 log.cc:824] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:07:23 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032443, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:07:23.591313 4307 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:07:23.591884 4307 tablet_bootstrap.cc:492] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Bootstrap complete.
I20260806 16:07:23.592516 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.576735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:51899)
0806 16:07:23.577225 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:23.577279 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:23.578178 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:23.579307 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:23.579315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:23.579761 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:07:23.580212 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:23.580226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.581985 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.581991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:23.583388 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:23.583404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.584438 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.584442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:23.584590 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.585252 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:07:23.585268 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:07:23.586887 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:07:23.589113 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:23.589119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:23.589131 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:23.589404 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:23.589774 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:23.589779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:23.589780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:23.589927 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:23.590536 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:07:23.590597 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:07:23.590896 (+ 299us) client_negotiation.cc:769] Sending connection context
0806 16:07:23.591144 (+ 248us) client_negotiation.cc:240] Negotiation successful
0806 16:07:23.591495 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":155,"threads_started":1}
I20260806 16:07:23.592504 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.576778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:51899 (local address 127.13.148.62:35479)
0806 16:07:23.577810 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:23.577827 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:23.577874 (+ 47us) server_negotiation.cc:407] Connection header received
0806 16:07:23.578625 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:23.578636 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:23.578928 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:23.579262 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:23.580510 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.581802 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.583551 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.584264 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.584454 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.587092 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:23.587097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:23.587108 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:23.587135 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:23.588943 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:23.589529 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:23.589538 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:23.589547 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:23.589669 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:23.590028 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:23.590031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:23.590033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:23.590380 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:23.590554 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:23.591291 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 16:07:23.591683 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":134,"threads_started":1}
I20260806 16:07:23.593484 4307 ts_tablet_manager.cc:1397] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20260806 16:07:23.595139 4465 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35479
I20260806 16:07:23.595517 4465 heartbeater.cc:464] Registering TS with master...
I20260806 16:07:23.596010 4465 heartbeater.cc:511] Master 127.13.148.62:35479 requested a full tablet report, sending...
I20260806 16:07:23.597038 4307 raft_consensus.cc:348] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:23.597761 4307 raft_consensus.cc:729] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68639b9520c94d70b832d7c52918c0dc, State: Initialized, Role: FOLLOWER
I20260806 16:07:23.598266 4307 consensus_queue.cc:260] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } }
I20260806 16:07:23.598770 4200 ts_manager.cc:194] Registered new tserver with Master: 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149)
I20260806 16:07:23.599319 4307 ts_tablet_manager.cc:1428] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 16:07:23.602365 4200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:51899
I20260806 16:07:23.605186 4465 heartbeater.cc:503] Master 127.13.148.62:35479 was elected leader, sending a full tablet report...
I20260806 16:07:23.608968 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.601704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.13.148.1:37149)
0806 16:07:23.602863 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:23.602877 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:23.603486 (+ 609us) server_negotiation.cc:407] Connection header received
0806 16:07:23.603724 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:23.603730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:23.604229 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:23.604393 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:23.604689 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.605850 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.607771 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.608295 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.608410 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.608508 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:23.608781 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 16:07:23.608863 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1053,"thread_start_us":120,"threads_started":1}
I20260806 16:07:23.608994 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.601577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37149 (local address 127.0.0.1:52242)
0806 16:07:23.603339 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:23.603393 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:23.603547 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:23.604407 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:23.604413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:23.604438 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:23.604558 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:23.604565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.605989 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.605992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:23.607613 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:23.607628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.608439 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.608442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:23.608556 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.608603 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 16:07:23.608723 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 16:07:23.608846 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1575,"thread_start_us":206,"threads_started":1}
I20260806 16:07:23.610340 4369 tablet_service.cc:1514] Processing DeleteTablet for tablet 569267b26eef4316b170702fdbfdadf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:07:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52242
I20260806 16:07:23.611282 4481 tablet_replica.cc:330] stopping tablet replica
I20260806 16:07:23.611593 4481 raft_consensus.cc:2229] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:07:23.611865 4481 raft_consensus.cc:2258] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:07:23.612808 4481 ts_tablet_manager.cc:1905] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:07:23.616333 4481 ts_tablet_manager.cc:1918] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:07:23.616418 4481 log.cc:1196] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal/wals/569267b26eef4316b170702fdbfdadf0
I20260806 16:07:23.616863 4481 ts_tablet_manager.cc:1939] T 569267b26eef4316b170702fdbfdadf0 P 68639b9520c94d70b832d7c52918c0dc: Deleting consensus metadata
I20260806 16:07:23.618662 4180 catalog_manager.cc:4841] TS 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149): tablet 569267b26eef4316b170702fdbfdadf0 (table test-table [id=cb60bb2c00f548089504faef2c6ac321]) successfully deleted
W20260806 16:07:23.708770 4476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:23.709064 4476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:23.709139 4476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:23.713112 4476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:23.713193 4476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:23.713310 4476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:07:23.718484 4476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:41565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=39933
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:23.719508 4476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:23.720525 4476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:23.730576 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:23.730773 4476 server_base.cc:1034] running on GCE node
W20260806 16:07:23.730616 4487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:23.730661 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:23.731331 4476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:23.731953 4476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:23.733127 4476 hybrid_clock.cc:648] HybridClock initialized: now 1786032443733110 us; error 26 us; skew 500 ppm
Aug 06 16:07:23 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032443, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:23.735999 4476 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:07:23.736802 4476 webserver.cc:460] Webserver started at http://127.13.148.2:39933/ using document root <none> and password file <none>
I20260806 16:07:23.737321 4476 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:23.737396 4476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:23.741324 4476 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:07:23.743530 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:23.744843 4476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:07:23.744995 4476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "816b527aff5e4c91b166164e715fdafe"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "be7606e789e122f9f62f6d3647f31e68"
server_key_iv: "9be8c8fc22e2aee082b090d3db5c793d"
server_key_version: "encryptionkey@0"
I20260806 16:07:23.745558 4476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:23.769361 4476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:23.773111 4476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:23.773320 4476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:23.773983 4476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:23.774868 4476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:07:23.774952 4476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:23.775044 4476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:07:23.775116 4476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:23.784369 4476 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:41565
I20260806 16:07:23.784421 4652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:41565 every 8 connection(s)
I20260806 16:07:23.785423 4476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:07:23.792682 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 4476
I20260806 16:07:23.793157 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:34683
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:35479
--webserver_port=41653
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:07:23 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032443, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:07:23.800448 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.787417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:57465 (local address 127.13.148.62:35479)
0806 16:07:23.787711 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:23.787715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:23.788490 (+ 775us) server_negotiation.cc:407] Connection header received
0806 16:07:23.789914 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:23.789917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:23.789982 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:23.790095 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:23.791337 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.791876 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.793092 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.793410 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.793539 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.796548 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:23.796551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:23.796554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:23.796580 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:23.798133 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:23.798815 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:23.798823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:23.798825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:23.798905 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:23.799288 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:23.799291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:23.799293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:23.799486 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:23.799597 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:23.800126 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 16:07:23.800267 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260806 16:07:23.801257 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.787829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:57465)
0806 16:07:23.788316 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:23.788361 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:23.789691 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:23.790287 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:23.790294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:23.790699 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:07:23.791172 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:23.791186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.792011 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.792015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:23.792963 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:23.792973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.793594 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.793598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:23.793780 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.794448 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:07:23.794523 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:07:23.796347 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:07:23.798274 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:23.798280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:23.798295 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:23.798697 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:23.799017 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:23.799021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:23.799023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:23.799179 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:23.799597 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:07:23.799605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:07:23.799880 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 16:07:23.800068 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 16:07:23.800413 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260806 16:07:23.802551 4654 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35479
I20260806 16:07:23.802860 4654 heartbeater.cc:464] Registering TS with master...
I20260806 16:07:23.803545 4654 heartbeater.cc:511] Master 127.13.148.62:35479 requested a full tablet report, sending...
I20260806 16:07:23.804962 4200 ts_manager.cc:194] Registered new tserver with Master: 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565)
I20260806 16:07:23.805653 4200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:57465
W20260806 16:07:23.911897 4659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:23.912170 4659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:23.912238 4659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:23.916494 4659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:23.916556 4659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:23.916638 4659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:07:23.922259 4659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:34683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=41653
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:23.923413 4659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:23.924477 4659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:23.934904 4671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:23.935151 4659 server_base.cc:1034] running on GCE node
W20260806 16:07:23.934900 4672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:23.934904 4674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:23.935791 4659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:23.936518 4659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:23.937722 4659 hybrid_clock.cc:648] HybridClock initialized: now 1786032443937700 us; error 33 us; skew 500 ppm
Aug 06 16:07:23 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032443, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:23.941067 4659 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:07:23.942024 4659 webserver.cc:460] Webserver started at http://127.13.148.3:41653/ using document root <none> and password file <none>
I20260806 16:07:23.942706 4659 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:23.942829 4659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:23.946844 4659 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:07:23.949208 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:23.950625 4659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:07:23.950799 4659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "67de039cef5f42fead896ffb9c4432f2"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "b629b417006f2e506080555993fe79f0"
server_key_iv: "d4dfbecc957a51726f9c1d160da74fed"
server_key_version: "encryptionkey@0"
I20260806 16:07:23.951370 4659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:23.967758 4659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:23.971031 4659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:23.971254 4659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:23.971872 4659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:23.972852 4659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:07:23.972930 4659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:23.973018 4659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:07:23.973068 4659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:23.982589 4659 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:34683
I20260806 16:07:23.982605 4832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:34683 every 8 connection(s)
I20260806 16:07:23.983626 4659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:07:23.992019 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 4659
Aug 06 16:07:23 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032443, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:07:23.998291 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.985728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:52569 (local address 127.13.148.62:35479)
0806 16:07:23.985863 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:23.985873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:23.986709 (+ 836us) server_negotiation.cc:407] Connection header received
0806 16:07:23.988057 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:23.988060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:23.988118 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:23.988194 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:23.989373 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.989913 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.991110 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.991337 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.991420 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.994281 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:23.994285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:23.994288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:23.994314 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:23.995903 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:23.996472 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:23.996479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:23.996482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:23.996569 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:23.997024 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:23.997026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:23.997028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:23.997200 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:23.997286 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:23.997990 (+ 704us) server_negotiation.cc:299] Negotiation successful
0806 16:07:23.998124 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:07:23.999300 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.986046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:52569)
0806 16:07:23.986526 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:23.986584 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:23.987842 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:23.988364 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:23.988370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:23.988753 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:07:23.989228 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:23.989241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.990027 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.990029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:23.990986 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:23.990996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.991468 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.991471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:23.991605 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.992278 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:07:23.992299 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:07:23.994110 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:07:23.996027 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:23.996036 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:23.996055 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:23.996364 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:23.996696 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:23.996699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:23.996700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:23.996916 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:23.997323 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:07:23.997333 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:07:23.997682 (+ 349us) client_negotiation.cc:769] Sending connection context
0806 16:07:23.997904 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 16:07:23.998243 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
I20260806 16:07:24.000674 4833 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35479
I20260806 16:07:24.000952 4833 heartbeater.cc:464] Registering TS with master...
I20260806 16:07:24.001328 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:23.992800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.13.148.62:35479)
0806 16:07:23.993079 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:23.993083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:23.993174 (+ 91us) server_negotiation.cc:407] Connection header received
0806 16:07:23.993296 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:23.993299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:23.993344 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:23.993442 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:23.994237 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.994992 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.995870 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:23.996221 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:23.996332 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:23.997794 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:23.997797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:23.997799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:23.997819 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:23.999362 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:23.999861 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:23.999864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:23.999865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:23.999971 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:24.000312 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:24.000315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:24.000317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:24.000534 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:24.000787 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:24.001018 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 16:07:24.001148 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20260806 16:07:24.001590 4833 heartbeater.cc:511] Master 127.13.148.62:35479 requested a full tablet report, sending...
I20260806 16:07:24.002810 4200 ts_manager.cc:194] Registered new tserver with Master: 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
I20260806 16:07:24.003579 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:07:24.003598 4200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:52569
I20260806 16:07:24.008103 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.004348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.13.148.62:35479)
0806 16:07:24.004443 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:24.004449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:24.004715 (+ 266us) server_negotiation.cc:407] Connection header received
0806 16:07:24.004887 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:24.004890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:24.004986 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:24.005106 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:07:24.005416 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.005904 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.006847 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.007089 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.007204 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.007552 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:07:24.007678 (+ 126us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 16:07:24.007952 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59652: token expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260806 16:07:24.008189 4839 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59652: token expired
W20260806 16:07:24.008529 4103 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.148.62:35479)
I20260806 16:07:24.016219 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.009000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.13.148.62:35479)
0806 16:07:24.009144 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:24.009147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:24.009162 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:07:24.009222 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:24.009224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:24.009259 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:24.009323 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:24.009919 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.010388 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.011230 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.011431 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.011510 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.012841 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:24.012844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:24.012846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:24.012872 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:24.014449 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:24.014873 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:24.014876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:24.014877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:24.014960 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:24.015301 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:24.015304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:24.015305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:24.015544 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:24.015659 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:24.015889 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 16:07:24.015994 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 16:07:24.018877 4200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:07:24.022101 4200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:07:24.030808 4369 tablet_service.cc:1467] Processing CreateTablet for tablet b7eedd4566f643abbabce8ed74adc6f1 (DEFAULT_TABLE table=test-table [id=873767975bad414c85bc6551964d65ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:07:24.031598 4369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7eedd4566f643abbabce8ed74adc6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:07:24.034722 4307 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Bootstrap starting.
I20260806 16:07:24.035722 4307 tablet_bootstrap.cc:654] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Neither blocks nor log segments found. Creating new log.
I20260806 16:07:24.035745 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.029571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41565 (local address 127.0.0.1:39398)
0806 16:07:24.030279 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:24.030301 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:24.030402 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:24.031304 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:24.031308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:24.031327 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:24.031432 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:24.031438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.033104 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.033108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:24.034235 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:24.034247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.035314 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.035318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:24.035412 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.035437 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 16:07:24.035490 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 16:07:24.035585 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553}
I20260806 16:07:24.036051 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.030660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.13.148.3:34683)
0806 16:07:24.031076 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:24.031087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:24.031112 (+ 25us) server_negotiation.cc:407] Connection header received
0806 16:07:24.031189 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:24.031195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:24.031378 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:24.031527 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:24.032012 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.033115 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.034138 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.035029 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.035146 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.035323 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:24.035646 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 16:07:24.035895 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":193,"threads_started":1}
I20260806 16:07:24.035745 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.029863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34683 (local address 127.0.0.1:33112)
0806 16:07:24.030265 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:24.030283 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:24.030391 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:24.031738 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:24.031741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:24.031759 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:24.031874 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:24.031882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.033241 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.033246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:24.034004 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:24.034012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.035377 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.035380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:24.035468 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.035495 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:07:24.035564 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 16:07:24.035614 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":1,"thread_start_us":133,"threads_started":1}
I20260806 16:07:24.036579 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.029781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.13.148.2:41565)
0806 16:07:24.030310 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:24.030337 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:24.030380 (+ 43us) server_negotiation.cc:407] Connection header received
0806 16:07:24.030830 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:24.030836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:24.031095 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:24.031289 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:24.031539 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.032976 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.034378 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.035166 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.035970 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.036143 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:24.036206 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 16:07:24.036426 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":179,"threads_started":1}
I20260806 16:07:24.037349 4307 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: No bootstrap required, opened a new log
I20260806 16:07:24.037480 4307 ts_tablet_manager.cc:1397] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 16:07:24.038251 4307 raft_consensus.cc:348] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.038378 4307 raft_consensus.cc:374] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:07:24.038416 4307 raft_consensus.cc:729] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68639b9520c94d70b832d7c52918c0dc, State: Initialized, Role: FOLLOWER
I20260806 16:07:24.038576 4307 consensus_queue.cc:260] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.038825 4307 ts_tablet_manager.cc:1428] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 16:07:24.039880 4566 tablet_service.cc:1467] Processing CreateTablet for tablet b7eedd4566f643abbabce8ed74adc6f1 (DEFAULT_TABLE table=test-table [id=873767975bad414c85bc6551964d65ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:07:24.039830 4747 tablet_service.cc:1467] Processing CreateTablet for tablet b7eedd4566f643abbabce8ed74adc6f1 (DEFAULT_TABLE table=test-table [id=873767975bad414c85bc6551964d65ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:07:24.041031 4566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7eedd4566f643abbabce8ed74adc6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:07:24.041041 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7eedd4566f643abbabce8ed74adc6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:07:24.047894 4852 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: Bootstrap starting.
I20260806 16:07:24.047894 4853 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: Bootstrap starting.
I20260806 16:07:24.050300 4853 tablet_bootstrap.cc:654] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: Neither blocks nor log segments found. Creating new log.
I20260806 16:07:24.050753 4852 tablet_bootstrap.cc:654] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: Neither blocks nor log segments found. Creating new log.
I20260806 16:07:24.051499 4853 log.cc:824] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:24.051499 4852 log.cc:824] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:24.053535 4852 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: No bootstrap required, opened a new log
I20260806 16:07:24.053759 4852 ts_tablet_manager.cc:1397] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 16:07:24.054045 4853 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: No bootstrap required, opened a new log
I20260806 16:07:24.054258 4853 ts_tablet_manager.cc:1397] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:07:24.057036 4852 raft_consensus.cc:348] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.057276 4852 raft_consensus.cc:374] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:07:24.057310 4852 raft_consensus.cc:729] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816b527aff5e4c91b166164e715fdafe, State: Initialized, Role: FOLLOWER
I20260806 16:07:24.057281 4853 raft_consensus.cc:348] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.057564 4853 raft_consensus.cc:374] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:07:24.057637 4853 raft_consensus.cc:729] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67de039cef5f42fead896ffb9c4432f2, State: Initialized, Role: FOLLOWER
I20260806 16:07:24.057811 4852 consensus_queue.cc:260] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.058054 4853 consensus_queue.cc:260] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.058651 4654 heartbeater.cc:503] Master 127.13.148.62:35479 was elected leader, sending a full tablet report...
I20260806 16:07:24.058933 4852 ts_tablet_manager.cc:1428] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260806 16:07:24.059001 4853 ts_tablet_manager.cc:1428] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:07:24.059340 4833 heartbeater.cc:503] Master 127.13.148.62:35479 was elected leader, sending a full tablet report...
W20260806 16:07:24.075601 4468 tablet.cc:2369] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:07:24.076301 4859 raft_consensus.cc:482] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:07:24.076467 4859 raft_consensus.cc:504] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.077521 4859 leader_election.cc:288] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565), 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149)
I20260806 16:07:24.081837 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.077910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:43433 (local address 127.13.148.2:41565)
0806 16:07:24.078045 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:24.078050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:24.078068 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:07:24.078169 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:24.078175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:24.078230 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:24.078321 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:24.078677 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.079422 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.080870 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.081258 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.081393 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.081445 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:24.081644 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 16:07:24.081724 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:07:24.081837 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.077782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41565 (local address 127.13.148.3:43433)
0806 16:07:24.077924 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:24.077941 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:24.078038 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:24.078349 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:24.078352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:24.078371 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:24.078550 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:24.078558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.079532 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.079535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:24.080730 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:24.080740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.081385 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.081389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:24.081498 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.081522 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:07:24.081629 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 16:07:24.081702 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 16:07:24.082304 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.077782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37149 (local address 127.13.148.3:53969)
0806 16:07:24.078376 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:24.078389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:24.078599 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:24.078914 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:24.078917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:24.078933 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:07:24.079054 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:24.079061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.079938 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.079942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:24.081012 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:24.081020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.081680 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.081684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:24.081785 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.081813 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 16:07:24.082053 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 16:07:24.082127 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":201,"thread_start_us":155,"threads_started":1}
I20260806 16:07:24.082304 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.077915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:53969 (local address 127.13.148.1:37149)
0806 16:07:24.078035 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:24.078039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:24.078465 (+ 426us) server_negotiation.cc:407] Connection header received
0806 16:07:24.078764 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:24.078767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:24.078819 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:24.078903 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:07:24.079190 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.079826 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.081134 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.081553 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.081684 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.081732 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:24.082104 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 16:07:24.082207 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:07:24.082695 4591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7eedd4566f643abbabce8ed74adc6f1" candidate_uuid: "67de039cef5f42fead896ffb9c4432f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816b527aff5e4c91b166164e715fdafe" is_pre_election: true
I20260806 16:07:24.083005 4591 raft_consensus.cc:2454] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67de039cef5f42fead896ffb9c4432f2 in term 0.
I20260806 16:07:24.083000 4399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7eedd4566f643abbabce8ed74adc6f1" candidate_uuid: "67de039cef5f42fead896ffb9c4432f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68639b9520c94d70b832d7c52918c0dc" is_pre_election: true
I20260806 16:07:24.083325 4399 raft_consensus.cc:2454] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67de039cef5f42fead896ffb9c4432f2 in term 0.
I20260806 16:07:24.083535 4686 leader_election.cc:302] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe; no voters:
I20260806 16:07:24.083820 4859 raft_consensus.cc:2781] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:07:24.083904 4859 raft_consensus.cc:482] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:07:24.083935 4859 raft_consensus.cc:3037] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:07:24.084995 4859 raft_consensus.cc:504] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.085371 4859 leader_election.cc:288] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [CANDIDATE]: Term 1 election: Requested vote from peers 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565), 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149)
I20260806 16:07:24.085732 4399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7eedd4566f643abbabce8ed74adc6f1" candidate_uuid: "67de039cef5f42fead896ffb9c4432f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68639b9520c94d70b832d7c52918c0dc"
I20260806 16:07:24.085750 4591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7eedd4566f643abbabce8ed74adc6f1" candidate_uuid: "67de039cef5f42fead896ffb9c4432f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816b527aff5e4c91b166164e715fdafe"
I20260806 16:07:24.085842 4399 raft_consensus.cc:3037] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:07:24.085881 4591 raft_consensus.cc:3037] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:07:24.086992 4399 raft_consensus.cc:2454] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67de039cef5f42fead896ffb9c4432f2 in term 1.
I20260806 16:07:24.087069 4591 raft_consensus.cc:2454] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67de039cef5f42fead896ffb9c4432f2 in term 1.
I20260806 16:07:24.087394 4686 leader_election.cc:302] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe; no voters:
I20260806 16:07:24.087580 4859 raft_consensus.cc:2781] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:07:24.087826 4859 raft_consensus.cc:686] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 1 LEADER]: Becoming Leader. State: Replica: 67de039cef5f42fead896ffb9c4432f2, State: Running, Role: LEADER
I20260806 16:07:24.088160 4859 consensus_queue.cc:237] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:24.091238 4200 catalog_manager.cc:5495] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "67de039cef5f42fead896ffb9c4432f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } health_report { overall_health: UNKNOWN } } }
I20260806 16:07:24.117219 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:24.113524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.13.148.3:34683)
0806 16:07:24.113673 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:24.113677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:24.113691 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:07:24.113848 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:24.113851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:24.113905 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:24.114032 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:07:24.114273 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.114885 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.115756 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:24.115961 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:24.116040 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:24.116316 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:07:24.116737 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:07:24.116830 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:24.116956 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 16:07:24.117006 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:07:24.123139 4399 raft_consensus.cc:1261] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Refusing update from remote peer 67de039cef5f42fead896ffb9c4432f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:07:24.123153 4591 raft_consensus.cc:1261] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 1 FOLLOWER]: Refusing update from remote peer 67de039cef5f42fead896ffb9c4432f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:07:24.123831 4861 consensus_queue.cc:1035] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:07:24.124006 4859 consensus_queue.cc:1035] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:07:24.130540 4868 mvcc.cc:205] Tried to move back new op lower bound from 7315588891121524736 to 7315588890987700224. Current Snapshot: MvccSnapshot[applied={T|T < 7315588891121524736}]
I20260806 16:07:24.132968 4870 mvcc.cc:205] Tried to move back new op lower bound from 7315588891121524736 to 7315588890987700224. Current Snapshot: MvccSnapshot[applied={T|T < 7315588891121524736}]
I20260806 16:07:24.133160 4866 mvcc.cc:205] Tried to move back new op lower bound from 7315588891121524736 to 7315588890987700224. Current Snapshot: MvccSnapshot[applied={T|T < 7315588891121524736 or (T in {7315588891121524736})}]
I20260806 16:07:24.138782 4200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:07:24.138953 4200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:07:24.141327 4200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Sending DeleteTablet for 3 replicas of tablet b7eedd4566f643abbabce8ed74adc6f1
I20260806 16:07:24.142005 4566 tablet_service.cc:1514] Processing DeleteTablet for tablet b7eedd4566f643abbabce8ed74adc6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:07:24 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:39398
I20260806 16:07:24.142114 4369 tablet_service.cc:1514] Processing DeleteTablet for tablet b7eedd4566f643abbabce8ed74adc6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:07:24 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:52242
I20260806 16:07:24.142124 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet b7eedd4566f643abbabce8ed74adc6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:07:24 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:33112
I20260806 16:07:24.142500 4882 tablet_replica.cc:330] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: stopping tablet replica
I20260806 16:07:24.142681 4882 raft_consensus.cc:2229] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:07:24.142735 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 4248
I20260806 16:07:24.142729 4886 tablet_replica.cc:330] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: stopping tablet replica
I20260806 16:07:24.142922 4882 raft_consensus.cc:2258] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:07:24.143070 4886 raft_consensus.cc:2229] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:07:24.145311 4886 raft_consensus.cc:2258] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:07:24.148420 4886 ts_tablet_manager.cc:1905] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:07:24.152843 4886 ts_tablet_manager.cc:1918] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:07:24.153293 4886 log.cc:1196] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/b7eedd4566f643abbabce8ed74adc6f1
I20260806 16:07:24.153777 4886 ts_tablet_manager.cc:1939] T b7eedd4566f643abbabce8ed74adc6f1 P 816b527aff5e4c91b166164e715fdafe: Deleting consensus metadata
I20260806 16:07:24.159560 4183 catalog_manager.cc:4841] TS 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565): tablet b7eedd4566f643abbabce8ed74adc6f1 (table test-table [id=873767975bad414c85bc6551964d65ad]) successfully deleted
I20260806 16:07:24.166352 4883 tablet_replica.cc:330] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: stopping tablet replica
I20260806 16:07:24.166829 4883 raft_consensus.cc:2229] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:07:24.167340 4883 raft_consensus.cc:2258] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:07:24.169081 4883 ts_tablet_manager.cc:1905] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:07:24.173014 4883 ts_tablet_manager.cc:1918] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:07:24.173138 4883 log.cc:1196] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/b7eedd4566f643abbabce8ed74adc6f1
I20260806 16:07:24.173503 4883 ts_tablet_manager.cc:1939] T b7eedd4566f643abbabce8ed74adc6f1 P 67de039cef5f42fead896ffb9c4432f2: Deleting consensus metadata
W20260806 16:07:24.174839 4180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:37149 (error 108)
I20260806 16:07:24.175041 4183 catalog_manager.cc:4841] TS 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683): tablet b7eedd4566f643abbabce8ed74adc6f1 (table test-table [id=873767975bad414c85bc6551964d65ad]) successfully deleted
W20260806 16:07:24.176033 4848 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:07:24.175725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37149 (local address 127.0.0.1:52256)
0806 16:07:24.175872 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:24.175943 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:37149: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 16:07:24.176203 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 4476
W20260806 16:07:24.186757 4180 catalog_manager.cc:4584] TS 68639b9520c94d70b832d7c52918c0dc (127.13.148.1:37149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7eedd4566f643abbabce8ed74adc6f1: Network error: Client connection negotiation failed: client connection to 127.13.148.1:37149: connect: Connection reset by peer (error 104)
I20260806 16:07:24.189615 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 4659
I20260806 16:07:24.200982 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 4158
I20260806 16:07:44.213872 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35479
--webserver_interface=127.13.148.62
--webserver_port=38247
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:07:44.332888 4889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:44.333192 4889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:44.333268 4889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:44.337358 4889 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 16:07:44.337412 4889 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 16:07:44.337428 4889 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 16:07:44.337442 4889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:07:44.337460 4889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:44.337515 4889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:44.337549 4889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:07:44.337570 4889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:07:44.342990 4889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:44.344152 4889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:44.345243 4889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:44.355428 4897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:44.355446 4896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:44.355965 4889 server_base.cc:1034] running on GCE node
W20260806 16:07:44.355439 4900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:44.356904 4889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:44.358107 4889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:44.359294 4889 hybrid_clock.cc:648] HybridClock initialized: now 1786032464359262 us; error 46 us; skew 500 ppm
Aug 06 16:07:44 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032464, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:44.362813 4889 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:07:44.363891 4889 webserver.cc:460] Webserver started at http://127.13.148.62:38247/ using document root <none> and password file <none>
I20260806 16:07:44.364542 4889 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:44.364661 4889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:44.369081 4889 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:07:44.371714 4914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:44.373490 4889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:07:44.373652 4889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "df0982a3339e4d30ae404cdf5ed3d524"
format_stamp: "Formatted at 2026-08-06 16:07:01 on dist-test-slave-7kzw"
server_key: "3e431af03247d91faddd965283ebbc47"
server_key_iv: "d70acbd4e5c5329959728e840e1a5221"
server_key_version: "encryptionkey@0"
I20260806 16:07:44.374655 4889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:44.390867 4889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:44.394534 4889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:44.394807 4889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:44.402493 4889 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35479
I20260806 16:07:44.402838 4994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35479 every 8 connection(s)
I20260806 16:07:44.403692 4889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:07:44.408380 4995 sys_catalog.cc:263] Verifying existing consensus state
I20260806 16:07:44.410131 4995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Bootstrap starting.
I20260806 16:07:44.413273 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 4889
I20260806 16:07:44.413771 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:37149
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:35479
--webserver_port=41759
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:07:44.429116 4995 log.cc:824] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:44.440626 4995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:07:44.441041 4995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Bootstrap complete.
I20260806 16:07:44.445130 4995 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:44.445503 4995 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0982a3339e4d30ae404cdf5ed3d524, State: Initialized, Role: FOLLOWER
I20260806 16:07:44.445991 4995 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:44.446110 4995 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:07:44.446197 4995 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:07:44.446322 4995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 2 FOLLOWER]: Advancing to term 3
I20260806 16:07:44.448271 4995 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:44.448618 4995 leader_election.cc:302] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0982a3339e4d30ae404cdf5ed3d524; no voters:
I20260806 16:07:44.448969 4995 leader_election.cc:288] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 16:07:44.449070 4999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 16:07:44.449276 4999 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [term 3 LEADER]: Becoming Leader. State: Replica: df0982a3339e4d30ae404cdf5ed3d524, State: Running, Role: LEADER
I20260806 16:07:44.449573 4999 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } }
I20260806 16:07:44.450263 4995 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:07:44.450552 4999 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "df0982a3339e4d30ae404cdf5ed3d524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } } }
I20260806 16:07:44.450668 4999 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: This master's current role is: LEADER
I20260806 16:07:44.450881 4999 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0982a3339e4d30ae404cdf5ed3d524. Latest consensus state: current_term: 3 leader_uuid: "df0982a3339e4d30ae404cdf5ed3d524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0982a3339e4d30ae404cdf5ed3d524" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35479 } } }
I20260806 16:07:44.450968 4999 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524 [sys.catalog]: This master's current role is: LEADER
I20260806 16:07:44.451207 5007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:07:44.456542 5007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:07:44.457445 5007 catalog_manager.cc:1259] Loaded cluster ID: 875e9463cd0e420b879dd25ae80bbd38
I20260806 16:07:44.457511 5007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:07:44.459975 5007 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:07:44.460705 5007 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Loaded TSK: 0
I20260806 16:07:44.460791 5007 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0982a3339e4d30ae404cdf5ed3d524: Loaded TSK: 1
F20260806 16:07:44.461676 5007 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fda3fcda9ad google::LogMessage::Fail() at ??:0
@ 0x7fda3fcdd88a google::LogMessage::SendToLog() at ??:0
@ 0x7fda3fcda4de google::LogMessage::Flush() at ??:0
@ 0x7fda3fcdba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fda4299e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fda42996f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fda4a6437bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fda4a61d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fda4a61e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fda4a65fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fda4a67e82b std::function<>::operator()() at ??:0
@ 0x7fda4a61db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fda4a61ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fda4a61b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fda4a65f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fda41f2c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fda41f2cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fda41f2e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fda41f1d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fda435616db start_thread at ??:0
@ 0x7fda3e7d861f clone at ??:0
W20260806 16:07:44.574899 4997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:44.575196 4997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:44.575242 4997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:44.579394 4997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:44.579455 4997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:44.579541 4997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:07:44.584930 4997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:37149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=41759
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:44.586018 4997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:44.587124 4997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:44.597105 5061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:44.597136 5058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:44.597388 4997 server_base.cc:1034] running on GCE node
W20260806 16:07:44.597457 5066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:44.597851 4997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:44.598553 4997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:44.599762 4997 hybrid_clock.cc:648] HybridClock initialized: now 1786032464599732 us; error 44 us; skew 500 ppm
Aug 06 16:07:44 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032464, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:44.602861 4997 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:07:44.603896 4997 webserver.cc:460] Webserver started at http://127.13.148.1:41759/ using document root <none> and password file <none>
I20260806 16:07:44.604473 4997 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:44.604570 4997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:44.608687 4997 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:07:44.611143 5075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:44.612457 4997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:07:44.612586 4997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "68639b9520c94d70b832d7c52918c0dc"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "d3addd862c35bad83742764be7333a53"
server_key_iv: "30eebee35c9fc1f42ee0422323474ccf"
server_key_version: "encryptionkey@0"
I20260806 16:07:44.613133 4997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:44.630167 4997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:44.633728 4997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:44.633950 4997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:44.634683 4997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:44.635878 5082 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 16:07:44.640185 4997 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 16:07:44.640257 4997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260806 16:07:44.640311 4997 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 16:07:44.642565 4997 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 16:07:44.642632 4997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 16:07:44.642895 5082 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Bootstrap starting.
I20260806 16:07:44.654132 4997 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:37149
I20260806 16:07:44.654488 5234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:37149 every 8 connection(s)
I20260806 16:07:44.655550 4997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:07:44.663980 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 4997
I20260806 16:07:44.664147 5082 log.cc:824] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Log is configured to *not* fsync() on all Append() calls
I20260806 16:07:44.664523 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:41565
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:35479
--webserver_port=39933
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:07:44.672178 5082 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 16:07:44.672670 5082 tablet_bootstrap.cc:492] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Bootstrap complete.
I20260806 16:07:44.673905 5082 ts_tablet_manager.cc:1397] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Time spent bootstrapping tablet: real 0.031s user 0.030s sys 0.000s
I20260806 16:07:44.676343 5082 raft_consensus.cc:348] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:44.676781 5082 raft_consensus.cc:729] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68639b9520c94d70b832d7c52918c0dc, State: Initialized, Role: FOLLOWER
I20260806 16:07:44.677277 5082 consensus_queue.cc:260] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:44.678225 5082 ts_tablet_manager.cc:1428] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260806 16:07:44.716229 5237 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:07:44.658039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:32783)
0806 16:07:44.658750 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:44.658812 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:44.659889 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:44.715389 (+ 55500us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":198,"threads_started":1}
W20260806 16:07:44.716753 5235 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:35479 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:35479: Client connection negotiation failed: client connection to 127.13.148.62:35479: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 16:07:44.717574 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:44.717333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:33597)
0806 16:07:44.717458 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:44.717521 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 16:07:44.718348 5237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:44.718159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:60085)
0806 16:07:44.718258 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:44.718299 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 16:07:44.718603 5235 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:07:44.796625 5240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:44.796900 5240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:44.796947 5240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:44.801043 5240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:44.801111 5240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:44.801170 5240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:07:44.806674 5240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:41565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=39933
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:44.807825 5240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:44.808897 5240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:44.819205 5250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:44.819272 5246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:44.819221 5248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:44.819370 5240 server_base.cc:1034] running on GCE node
I20260806 16:07:44.819953 5240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:44.820647 5240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:44.821815 5240 hybrid_clock.cc:648] HybridClock initialized: now 1786032464821801 us; error 30 us; skew 500 ppm
Aug 06 16:07:44 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032464, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:44.824963 5240 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:07:44.825867 5240 webserver.cc:460] Webserver started at http://127.13.148.2:39933/ using document root <none> and password file <none>
I20260806 16:07:44.826431 5240 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:44.826550 5240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:44.830323 5240 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:07:44.832517 5257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:44.833773 5240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:07:44.833922 5240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "816b527aff5e4c91b166164e715fdafe"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "be7606e789e122f9f62f6d3647f31e68"
server_key_iv: "9be8c8fc22e2aee082b090d3db5c793d"
server_key_version: "encryptionkey@0"
I20260806 16:07:44.834615 5240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:44.855149 5240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:44.858814 5240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:44.859047 5240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:44.859716 5240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:44.860664 5240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:07:44.860718 5240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:44.860795 5240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:07:44.860833 5240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:44.870606 5240 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:41565
I20260806 16:07:44.870646 5426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:41565 every 8 connection(s)
I20260806 16:07:44.871747 5240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:07:44.875126 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 5240
I20260806 16:07:44.875643 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:34683
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:35479
--webserver_port=41653
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:07:44.876761 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:44.874359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:47195)
0806 16:07:44.875018 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:44.875138 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":141,"threads_started":1}
W20260806 16:07:44.877369 5427 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:35479 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:35479: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
I20260806 16:07:44.878321 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:44.878001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:34363)
0806 16:07:44.878198 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:44.878255 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 16:07:44.879040 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:44.878807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:40373)
0806 16:07:44.878927 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:44.878980 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 16:07:44.879213 5427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 16:07:44.993417 5430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:07:44.993778 5430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:07:44.993865 5430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:07:44.997888 5430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:07:44.998003 5430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:07:44.998102 5430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:07:45.003454 5430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:34683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=41653
--tserver_master_addrs=127.13.148.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:07:45.004655 5430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:07:45.005760 5430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:07:45.015700 5439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:45.015700 5437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:07:45.015722 5435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:07:45.015776 5430 server_base.cc:1034] running on GCE node
I20260806 16:07:45.016337 5430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:07:45.017100 5430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:07:45.018296 5430 hybrid_clock.cc:648] HybridClock initialized: now 1786032465018287 us; error 33 us; skew 500 ppm
Aug 06 16:07:45 dist-test-slave-7kzw krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032465, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:07:45.021266 5430 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:07:45.022233 5430 webserver.cc:460] Webserver started at http://127.13.148.3:41653/ using document root <none> and password file <none>
I20260806 16:07:45.022889 5430 fs_manager.cc:359] Metadata directory not provided
I20260806 16:07:45.022990 5430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:07:45.026994 5430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:07:45.029155 5450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:45.030547 5430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:07:45.030697 5430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "67de039cef5f42fead896ffb9c4432f2"
format_stamp: "Formatted at 2026-08-06 16:07:02 on dist-test-slave-7kzw"
server_key: "b629b417006f2e506080555993fe79f0"
server_key_iv: "d4dfbecc957a51726f9c1d160da74fed"
server_key_version: "encryptionkey@0"
I20260806 16:07:45.031272 5430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:07:45.060715 5430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:07:45.064028 5430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:07:45.064267 5430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:07:45.064898 5430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:07:45.065779 5430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:07:45.065856 5430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:45.065933 5430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:07:45.065977 5430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:07:45.076025 5430 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:34683
I20260806 16:07:45.076067 5610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:34683 every 8 connection(s)
I20260806 16:07:45.077067 5430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:07:45.081167 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:45.079312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:43419)
0806 16:07:45.079832 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:45.079952 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":129,"threads_started":1}
W20260806 16:07:45.081722 5612 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:35479 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:35479: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
I20260806 16:07:45.082453 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:45.082148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:32887)
0806 16:07:45.082340 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:45.082393 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 16:07:45.083149 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:45.082937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:58743)
0806 16:07:45.083053 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:45.083100 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 16:07:45.083385 5612 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 16:07:45.084717 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 5430
I20260806 16:07:45.719664 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:45.719193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:50703)
0806 16:07:45.719483 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:45.719543 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260806 16:07:45.880467 5620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:45.879954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:39309)
0806 16:07:45.880277 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:45.880347 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":80,"threads_started":1}
I20260806 16:07:46.084504 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:46.084042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:41209)
0806 16:07:46.084327 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:46.084388 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260806 16:07:46.457815 5623 raft_consensus.cc:482] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:07:46.458159 5623 raft_consensus.cc:504] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:46.459591 5623 leader_election.cc:288] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565), 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
Aug 06 16:07:46 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032464, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.3@KRBTEST.COM
Aug 06 16:07:46 dist-test-slave-7kzw krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032464, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.2@KRBTEST.COM
I20260806 16:07:46.476553 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:46.460423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34683 (local address 127.13.148.1:46949)
0806 16:07:46.460964 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:46.460976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:46.461091 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:46.462726 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:46.462730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:46.463256 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:07:46.463779 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:46.463804 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:46.465224 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:46.465227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:46.466227 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:46.466238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:46.467105 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:46.467109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:46.467391 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:46.468202 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:07:46.468215 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:07:46.469909 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:07:46.473026 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:46.473033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:46.473048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:46.473556 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:46.474092 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:46.474094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:46.474096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:46.474306 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:46.475208 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:07:46.475215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:07:46.475608 (+ 393us) client_negotiation.cc:769] Sending connection context
0806 16:07:46.475840 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 16:07:46.476398 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":455,"spinlock_wait_cycles":64000,"thread_start_us":71,"threads_started":1}
I20260806 16:07:46.476580 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:46.460890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:46949 (local address 127.13.148.3:34683)
0806 16:07:46.461245 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:46.461266 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:46.461300 (+ 34us) server_negotiation.cc:407] Connection header received
0806 16:07:46.462093 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:46.462101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:46.462410 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:46.462783 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:46.463969 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:46.465049 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:46.466367 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:46.466949 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:46.467143 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:46.470185 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:46.470194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:46.470214 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:46.470256 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:46.472898 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:46.473724 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:46.473731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:46.473737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:46.473930 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:46.474424 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:46.474427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:46.474430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:46.475047 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:46.475207 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:46.476010 (+ 803us) server_negotiation.cc:299] Negotiation successful
0806 16:07:46.476369 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
I20260806 16:07:46.476588 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:46.460118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:40435 (local address 127.13.148.2:41565)
0806 16:07:46.460610 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 16:07:46.460629 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:07:46.460684 (+ 55us) server_negotiation.cc:407] Connection header received
0806 16:07:46.461708 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:07:46.461716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:07:46.462202 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:07:46.462554 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:07:46.463973 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:46.465049 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:46.466366 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:07:46.466935 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:07:46.467135 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:46.470552 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:07:46.470558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:07:46.470570 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:07:46.470601 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:07:46.472888 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:46.473724 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:46.473731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:46.473737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:46.473929 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:07:46.474424 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:07:46.474427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:07:46.474429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:07:46.475047 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:07:46.475207 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 16:07:46.476010 (+ 803us) server_negotiation.cc:299] Negotiation successful
0806 16:07:46.476369 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":98,"threads_started":1}
I20260806 16:07:46.476553 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:46.459918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41565 (local address 127.13.148.1:40435)
0806 16:07:46.460385 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:46.460403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:07:46.460812 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:07:46.462627 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:07:46.462634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:07:46.463244 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:07:46.463778 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:46.463794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:46.465213 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:46.465222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:46.466226 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:07:46.466237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:07:46.467074 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:07:46.467078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:07:46.467391 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:07:46.468145 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:07:46.468164 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:07:46.470310 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:07:46.473026 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:46.473033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:46.473048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:46.473556 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:46.474073 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:07:46.474080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:07:46.474082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:07:46.474306 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:07:46.475208 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:07:46.475215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:07:46.475608 (+ 393us) client_negotiation.cc:769] Sending connection context
0806 16:07:46.475840 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 16:07:46.476326 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":254,"spinlock_wait_cycles":17408,"thread_start_us":145,"threads_started":1}
W20260806 16:07:46.479750 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
W20260806 16:07:46.480018 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
I20260806 16:07:46.480113 5079 leader_election.cc:302] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68639b9520c94d70b832d7c52918c0dc; no voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe
I20260806 16:07:46.480440 5623 raft_consensus.cc:2726] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:07:46.720901 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:46.720528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:43207)
0806 16:07:46.720716 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:46.720785 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 16:07:46.881899 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:46.881422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:56819)
0806 16:07:46.881718 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:46.881776 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":81,"threads_started":1}
I20260806 16:07:47.085783 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:47.085365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:39239)
0806 16:07:47.085616 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:47.085682 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260806 16:07:47.722260 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:47.721735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:54717)
0806 16:07:47.722068 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:47.722156 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
I20260806 16:07:47.883355 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:47.882901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:53523)
0806 16:07:47.883166 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:47.883230 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260806 16:07:48.087194 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:48.086676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:39685)
0806 16:07:48.087033 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:48.087093 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260806 16:07:48.200305 5636 raft_consensus.cc:482] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:07:48.200434 5636 raft_consensus.cc:504] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:48.200909 5636 leader_election.cc:288] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565), 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
W20260806 16:07:48.201828 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
W20260806 16:07:48.201977 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
I20260806 16:07:48.202042 5079 leader_election.cc:302] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68639b9520c94d70b832d7c52918c0dc; no voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe
I20260806 16:07:48.202271 5636 raft_consensus.cc:2726] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:07:48.723556 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:48.723120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:56071)
0806 16:07:48.723397 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:48.723453 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260806 16:07:48.884869 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:48.884420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:45573)
0806 16:07:48.884695 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:48.884771 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260806 16:07:49.088563 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:49.088081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:58541)
0806 16:07:49.088366 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:49.088432 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260806 16:07:49.725065 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:49.724556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:39779)
0806 16:07:49.724896 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:49.724957 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":102,"threads_started":1}
I20260806 16:07:49.886548 5643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:49.885832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:43215)
0806 16:07:49.886255 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:49.886335 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":150,"threads_started":1}
I20260806 16:07:50.090039 5644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:50.089523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:44381)
0806 16:07:50.089866 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:50.089929 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260806 16:07:50.726315 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:50.725866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:44781)
0806 16:07:50.726156 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:50.726215 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
I20260806 16:07:50.754983 5647 raft_consensus.cc:482] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:07:50.755163 5647 raft_consensus.cc:504] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:50.755669 5647 leader_election.cc:288] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565), 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
W20260806 16:07:50.756465 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
W20260806 16:07:50.756592 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
I20260806 16:07:50.756630 5079 leader_election.cc:302] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68639b9520c94d70b832d7c52918c0dc; no voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe
I20260806 16:07:50.756788 5647 raft_consensus.cc:2726] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:07:50.887914 5648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:50.887446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:47819)
0806 16:07:50.887744 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:50.887807 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":75,"threads_started":1}
I20260806 16:07:51.091347 5649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:51.090904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:39831)
0806 16:07:51.091184 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:51.091253 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260806 16:07:51.727558 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:51.727100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:53713)
0806 16:07:51.727399 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:51.727458 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":79,"threads_started":1}
I20260806 16:07:51.889353 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:51.888845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:44761)
0806 16:07:51.889191 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:51.889249 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":89,"threads_started":1}
I20260806 16:07:52.092681 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:52.092158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:56293)
0806 16:07:52.092482 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:52.092545 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":77,"threads_started":1}
I20260806 16:07:52.728917 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:52.728410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:34291)
0806 16:07:52.728742 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:52.728803 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":81,"threads_started":1}
I20260806 16:07:52.890967 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:52.890396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:40633)
0806 16:07:52.890765 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:52.890826 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":183,"threads_started":1}
I20260806 16:07:53.094127 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:53.093643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:54317)
0806 16:07:53.093952 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:53.094020 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260806 16:07:53.602275 5657 raft_consensus.cc:482] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:07:53.602491 5657 raft_consensus.cc:504] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:53.603106 5657 leader_election.cc:288] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565), 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
W20260806 16:07:53.603966 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
W20260806 16:07:53.604138 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
I20260806 16:07:53.604199 5079 leader_election.cc:302] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68639b9520c94d70b832d7c52918c0dc; no voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe
I20260806 16:07:53.604362 5657 raft_consensus.cc:2726] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:07:53.730340 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:53.729820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:33701)
0806 16:07:53.730163 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:53.730231 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
I20260806 16:07:53.892271 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:53.891830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:35483)
0806 16:07:53.892111 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:53.892170 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260806 16:07:54.095549 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:54.095061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:34963)
0806 16:07:54.095383 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:54.095446 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":115,"threads_started":1}
I20260806 16:07:54.731726 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:54.731253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:50501)
0806 16:07:54.731547 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:54.731610 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260806 16:07:54.893712 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:54.893217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:49663)
0806 16:07:54.893526 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:54.893588 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260806 16:07:55.097025 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:55.096543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:38103)
0806 16:07:55.096862 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:55.096926 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20260806 16:07:55.733055 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:55.732598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:46441)
0806 16:07:55.732884 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:55.732946 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260806 16:07:55.895134 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:55.894681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:36809)
0806 16:07:55.894968 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:55.895028 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260806 16:07:56.098421 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:56.097906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:42669)
0806 16:07:56.098242 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:56.098310 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260806 16:07:56.734298 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:56.733861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:55231)
0806 16:07:56.734112 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:56.734165 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260806 16:07:56.896622 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:56.896109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:57177)
0806 16:07:56.896443 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:56.896507 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260806 16:07:57.099941 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:57.099453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:45873)
0806 16:07:57.099779 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:57.099838 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":133,"threads_started":1}
I20260806 16:07:57.735846 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:57.735363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:48389)
0806 16:07:57.735679 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:57.735743 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260806 16:07:57.898183 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:57.897700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:57517)
0806 16:07:57.898004 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:57.898071 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260806 16:07:58.101322 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:58.100855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:47341)
0806 16:07:58.101153 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:58.101220 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20260806 16:07:58.737316 5675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:58.736797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:44097)
0806 16:07:58.737148 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:58.737212 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":169,"threads_started":1}
I20260806 16:07:58.899600 5676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:58.899064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:45165)
0806 16:07:58.899387 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:58.899456 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260806 16:07:58.980715 5678 raft_consensus.cc:482] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:07:58.980873 5678 raft_consensus.cc:504] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "816b527aff5e4c91b166164e715fdafe" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41565 } } peers { permanent_uuid: "67de039cef5f42fead896ffb9c4432f2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34683 } } peers { permanent_uuid: "68639b9520c94d70b832d7c52918c0dc" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37149 } }
I20260806 16:07:58.981297 5678 leader_election.cc:288] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565), 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683)
W20260806 16:07:58.982307 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 816b527aff5e4c91b166164e715fdafe (127.13.148.2:41565): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
W20260806 16:07:58.982616 5079 leader_election.cc:341] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67de039cef5f42fead896ffb9c4432f2 (127.13.148.3:34683): Not found: Tablet not found: b7eedd4566f643abbabce8ed74adc6f1
I20260806 16:07:58.982708 5079 leader_election.cc:302] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68639b9520c94d70b832d7c52918c0dc; no voters: 67de039cef5f42fead896ffb9c4432f2, 816b527aff5e4c91b166164e715fdafe
I20260806 16:07:58.982899 5678 raft_consensus.cc:2726] T b7eedd4566f643abbabce8ed74adc6f1 P 68639b9520c94d70b832d7c52918c0dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 16:07:59.102880 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:59.102289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:59979)
0806 16:07:59.102704 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:59.102772 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":114,"threads_started":1}
I20260806 16:07:59.738880 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:59.738376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.1:59141)
0806 16:07:59.738720 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:59.738782 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
I20260806 16:07:59.901298 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:07:59.900805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.2:44843)
0806 16:07:59.901103 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 16:07:59.901189 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":120,"threads_started":1}
I20260806 16:08:00.093968 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 4997
I20260806 16:08:00.104552 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:00.104046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35479 (local address 127.13.148.3:38061)
0806 16:08:00.104374 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:00.104444 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:35479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
I20260806 16:08:00.108906 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 5240
I20260806 16:08:00.121701 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 5430
2026-08-06T16:08:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59674 ms)
[----------] 30 tests from SecurityITest (252524 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:00 dist-test-slave-7kzw krb5kdc[5685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:00 dist-test-slave-7kzw krb5kdc[5685](info): set up 2 sockets
Aug 06 16:08:00 dist-test-slave-7kzw krb5kdc[5685](info): commencing operation
krb5kdc: starting...
W20260806 16:08:01.413077 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:01 dist-test-slave-7kzw krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:01.549203 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46063
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:01.669807 5705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:01.670192 5705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:01.670279 5705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:01.674443 5705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:01.674568 5705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:01.674616 5705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:01.674654 5705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:01.674687 5705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:01.680121 5705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:01.681205 5705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:01.682257 5705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:08:01.691419 5705 server_base.cc:1034] running on GCE node
W20260806 16:08:01.691442 5716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:01.691376 5714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:01.691425 5713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:01.692246 5705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:01.693303 5705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:01.694556 5705 hybrid_clock.cc:648] HybridClock initialized: now 1786032481694534 us; error 31 us; skew 500 ppm
Aug 06 16:08:01 dist-test-slave-7kzw krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032481, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:01.697546 5705 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:01.698419 5705 webserver.cc:460] Webserver started at http://127.13.148.62:39209/ using document root <none> and password file <none>
I20260806 16:08:01.699025 5705 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:01.699100 5705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:01.699281 5705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:01.701284 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "165ed314f8d44df9a19e498e65c1f0d4"
format_stamp: "Formatted at 2026-08-06 16:08:01 on dist-test-slave-7kzw"
server_key: "66a782ec8584175edbe2e9e8adbcf163"
server_key_iv: "364f1722b6a128398c09c4285080c56e"
server_key_version: "encryptionkey@0"
I20260806 16:08:01.701805 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "165ed314f8d44df9a19e498e65c1f0d4"
format_stamp: "Formatted at 2026-08-06 16:08:01 on dist-test-slave-7kzw"
server_key: "66a782ec8584175edbe2e9e8adbcf163"
server_key_iv: "364f1722b6a128398c09c4285080c56e"
server_key_version: "encryptionkey@0"
I20260806 16:08:01.705791 5705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:08:01.708421 5730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:01.709776 5705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:08:01.710011 5705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "165ed314f8d44df9a19e498e65c1f0d4"
format_stamp: "Formatted at 2026-08-06 16:08:01 on dist-test-slave-7kzw"
server_key: "66a782ec8584175edbe2e9e8adbcf163"
server_key_iv: "364f1722b6a128398c09c4285080c56e"
server_key_version: "encryptionkey@0"
I20260806 16:08:01.710132 5705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:01.733155 5705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:01.736759 5705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:01.736972 5705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:01.744576 5705 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46063
I20260806 16:08:01.744644 5803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46063 every 8 connection(s)
I20260806 16:08:01.745610 5705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:01.746306 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 5705
I20260806 16:08:01.746469 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:01.746810 13904 external_mini_cluster.cc:1428] Setting key 4c8da8c6afae3d74f1c8c3c28796db49
I20260806 16:08:01.749322 5804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:01.756664 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4: Bootstrap starting.
Aug 06 16:08:01 dist-test-slave-7kzw krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:01.759456 5804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:01.760241 5804 log.cc:824] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:01.762734 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4: No bootstrap required, opened a new log
I20260806 16:08:01.763473 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:01.748380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.13.148.62:46063)
0806 16:08:01.748969 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:01.748992 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:01.749040 (+ 48us) server_negotiation.cc:407] Connection header received
0806 16:08:01.749969 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:01.749996 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:01.750436 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:01.750860 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:01.751541 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:01.753318 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:01.754215 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:01.755046 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:01.755219 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:01.758283 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:01.758288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:01.758300 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:01.758327 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:01.760220 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:01.760792 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:01.760799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:01.760804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:01.760963 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:01.761366 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:01.761369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:01.761370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:01.761867 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:01.762008 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:01.762340 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 16:08:01.762717 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":205,"threads_started":1}
I20260806 16:08:01.766574 5804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165ed314f8d44df9a19e498e65c1f0d4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46063 } }
I20260806 16:08:01.766813 5804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:01.766904 5804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165ed314f8d44df9a19e498e65c1f0d4, State: Initialized, Role: FOLLOWER
I20260806 16:08:01.767335 5804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165ed314f8d44df9a19e498e65c1f0d4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46063 } }
I20260806 16:08:01.767493 5804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:01.767580 5804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:01.767696 5804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:01.768566 5804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165ed314f8d44df9a19e498e65c1f0d4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46063 } }
I20260806 16:08:01.768901 5804 leader_election.cc:302] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165ed314f8d44df9a19e498e65c1f0d4; no voters:
I20260806 16:08:01.769228 5804 leader_election.cc:288] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:01.769353 5811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:01.769703 5811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [term 1 LEADER]: Becoming Leader. State: Replica: 165ed314f8d44df9a19e498e65c1f0d4, State: Running, Role: LEADER
I20260806 16:08:01.770421 5811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165ed314f8d44df9a19e498e65c1f0d4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46063 } }
I20260806 16:08:01.770709 5804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:01.772382 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165ed314f8d44df9a19e498e65c1f0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165ed314f8d44df9a19e498e65c1f0d4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46063 } } }
I20260806 16:08:01.772357 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165ed314f8d44df9a19e498e65c1f0d4. Latest consensus state: current_term: 1 leader_uuid: "165ed314f8d44df9a19e498e65c1f0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165ed314f8d44df9a19e498e65c1f0d4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46063 } } }
I20260806 16:08:01.772522 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:01.772522 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:01.772991 5829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:01.776463 5829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:01.782351 5829 catalog_manager.cc:1347] Generated new cluster ID: 2550211e606343aea7804e30881af283
I20260806 16:08:01.782446 5829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:01.799480 5829 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:01.800578 5829 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:01.806654 5829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4: Generated new TSK 0
I20260806 16:08:01.807421 5829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:08:01.847586 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46063
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:01.985760 5840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:01.986023 5840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:01.986064 5840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:01.990023 5840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:01.990087 5840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:01.990144 5840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:08:01.995227 5840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:01.996253 5840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:01.997205 5840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:02.007786 5847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:02.008203 5846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:02.008232 5851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:02.008392 5840 server_base.cc:1034] running on GCE node
I20260806 16:08:02.008836 5840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:02.009443 5840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:02.010658 5840 hybrid_clock.cc:648] HybridClock initialized: now 1786032482010621 us; error 45 us; skew 500 ppm
Aug 06 16:08:02 dist-test-slave-7kzw krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032482, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:02.013811 5840 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:08:02.014912 5840 webserver.cc:460] Webserver started at http://127.13.148.1:45503/ using document root <none> and password file <none>
I20260806 16:08:02.015494 5840 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:02.015558 5840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:02.015799 5840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:02.017835 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "c1e33c5478e94cac965ed665d3abeaaa"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "8a85c5282045b3cc46d6d99bc518f7bb"
server_key_iv: "3e1d91838ad69893994f710186ad498e"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.018422 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "c1e33c5478e94cac965ed665d3abeaaa"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "8a85c5282045b3cc46d6d99bc518f7bb"
server_key_iv: "3e1d91838ad69893994f710186ad498e"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.022351 5840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 16:08:02.024860 5861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.026175 5840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:02.026304 5840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "c1e33c5478e94cac965ed665d3abeaaa"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "8a85c5282045b3cc46d6d99bc518f7bb"
server_key_iv: "3e1d91838ad69893994f710186ad498e"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.026413 5840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:02.061239 5840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:02.064844 5840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:02.065073 5840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:02.065714 5840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:02.066597 5840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:02.066656 5840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.066735 5840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:02.066766 5840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.076167 5840 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:45579
I20260806 16:08:02.076202 6013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:45579 every 8 connection(s)
I20260806 16:08:02.077098 5840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:08:02.086764 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 5840
I20260806 16:08:02.086931 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:08:02.087234 13904 external_mini_cluster.cc:1428] Setting key a0afef020a6f99e66cfcf3b1ef32dd91
Aug 06 16:08:02 dist-test-slave-7kzw krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032482, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:02.092195 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.078940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:46857 (local address 127.13.148.62:46063)
0806 16:08:02.079076 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:02.079080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:02.079871 (+ 791us) server_negotiation.cc:407] Connection header received
0806 16:08:02.081271 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:02.081274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:02.081334 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:02.081447 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:02.082645 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.083182 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.084671 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.085100 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.085208 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.088031 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:02.088035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:02.088038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:02.088066 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:02.089692 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:02.090349 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:02.090351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:02.090352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:02.090427 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:02.090894 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:02.090896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:02.090897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:02.091143 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:02.091269 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:02.091789 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 16:08:02.091938 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:08:02.092859 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.079274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46063 (local address 127.13.148.1:46857)
0806 16:08:02.079721 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:02.079764 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:02.081048 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:02.081626 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:02.081633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:02.081984 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:02.082441 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.082453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.083349 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.083352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.084559 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.084570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.085254 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.085258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.085433 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.086051 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:02.086067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:02.087827 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:02.089819 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:02.089828 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:02.089847 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:02.090240 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:02.090539 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:02.090545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:02.090550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:02.090794 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:02.091287 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:02.091295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:02.091548 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 16:08:02.091718 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 16:08:02.091972 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260806 16:08:02.094146 6015 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46063
I20260806 16:08:02.094429 6015 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:02.095077 6015 heartbeater.cc:511] Master 127.13.148.62:46063 requested a full tablet report, sending...
I20260806 16:08:02.096777 5756 ts_manager.cc:194] Registered new tserver with Master: c1e33c5478e94cac965ed665d3abeaaa (127.13.148.1:45579)
I20260806 16:08:02.097930 5756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:46857
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:08:02.118285 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46063
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:02.231398 6020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:02.231684 6020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:02.231761 6020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:02.235868 6020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:02.235960 6020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:02.236053 6020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:08:02.241430 6020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:02.242650 6020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:02.243719 6020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:02.253607 6027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:02.253679 6026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:02.253607 6029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:02.253717 6020 server_base.cc:1034] running on GCE node
I20260806 16:08:02.254333 6020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:02.255019 6020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:02.256225 6020 hybrid_clock.cc:648] HybridClock initialized: now 1786032482256197 us; error 37 us; skew 500 ppm
Aug 06 16:08:02 dist-test-slave-7kzw krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032482, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:02.259205 6020 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:08:02.260231 6020 webserver.cc:460] Webserver started at http://127.13.148.2:46281/ using document root <none> and password file <none>
I20260806 16:08:02.260836 6020 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:02.260916 6020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:02.261183 6020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:02.263609 6020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "f7e70f553ab24fb9a6bb37a0c58b302d"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "8f6c6a01c64461dc2e6d4c374f2f8dfa"
server_key_iv: "78607a4702653e66eac819f9fe481590"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.264231 6020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "f7e70f553ab24fb9a6bb37a0c58b302d"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "8f6c6a01c64461dc2e6d4c374f2f8dfa"
server_key_iv: "78607a4702653e66eac819f9fe481590"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.268417 6020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:08:02.271402 6038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.273023 6020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:08:02.273190 6020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "f7e70f553ab24fb9a6bb37a0c58b302d"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "8f6c6a01c64461dc2e6d4c374f2f8dfa"
server_key_iv: "78607a4702653e66eac819f9fe481590"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.273332 6020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:02.285136 6020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:02.288290 6020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:02.288509 6020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:02.289130 6020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:02.289978 6020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:02.290051 6020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.290128 6020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:02.290179 6020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.299566 6020 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:39503
I20260806 16:08:02.299613 6199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:39503 every 8 connection(s)
I20260806 16:08:02.300529 6020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:08:02.305455 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 6020
I20260806 16:08:02.305605 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:08:02.305934 13904 external_mini_cluster.cc:1428] Setting key a546402bec6e4bf60447661d6505a7d0
Aug 06 16:08:02 dist-test-slave-7kzw krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032482, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:02.317077 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.302522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:48263 (local address 127.13.148.62:46063)
0806 16:08:02.302690 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:02.302694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:02.303487 (+ 793us) server_negotiation.cc:407] Connection header received
0806 16:08:02.304904 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:02.304907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:02.304981 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:02.305143 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:02.306818 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.307572 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.308916 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.309342 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.309484 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.312788 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:02.312791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:02.312793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:02.312820 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:02.314778 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:02.315367 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:02.315371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:02.315373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:02.315462 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:02.315854 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:02.315858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:02.315860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:02.316060 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:02.316179 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:02.316742 (+ 563us) server_negotiation.cc:299] Negotiation successful
0806 16:08:02.316880 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260806 16:08:02.317999 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.302846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46063 (local address 127.13.148.2:48263)
0806 16:08:02.303314 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:02.303357 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:02.304675 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:02.305367 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:02.305374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:02.305909 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:02.306628 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.306652 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.307731 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.307734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.308774 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.308785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.309537 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.309541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.309834 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.310643 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:02.310673 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:02.312581 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:02.314909 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:02.314915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:02.314931 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:02.315243 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:02.315573 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:02.315576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:02.315578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:02.315739 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:02.316186 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:02.316196 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:02.316463 (+ 267us) client_negotiation.cc:769] Sending connection context
0806 16:08:02.316667 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 16:08:02.317028 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":103,"threads_started":1}
I20260806 16:08:02.319334 6200 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46063
I20260806 16:08:02.319574 6200 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:02.320087 6200 heartbeater.cc:511] Master 127.13.148.62:46063 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260806 16:08:02.321358 5756 ts_manager.cc:194] Registered new tserver with Master: f7e70f553ab24fb9a6bb37a0c58b302d (127.13.148.2:39503)
I20260806 16:08:02.322060 5756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:48263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:08:02.338377 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46063
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:02.458689 6208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:02.459012 6208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:02.459115 6208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:02.463330 6208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:02.463415 6208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:02.463534 6208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:08:02.468921 6208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:02.470052 6208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:02.471184 6208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:02.481492 6213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:02.481493 6214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:02.481493 6218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:02.481551 6208 server_base.cc:1034] running on GCE node
I20260806 16:08:02.482316 6208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:02.483057 6208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:02.484295 6208 hybrid_clock.cc:648] HybridClock initialized: now 1786032482484279 us; error 27 us; skew 500 ppm
Aug 06 16:08:02 dist-test-slave-7kzw krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032482, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:02.487596 6208 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:08:02.488705 6208 webserver.cc:460] Webserver started at http://127.13.148.3:39151/ using document root <none> and password file <none>
I20260806 16:08:02.489341 6208 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:02.489444 6208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:02.489698 6208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:02.491967 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "088f7af6aa4d46b999db5840d601d33c"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "03f63b854b2692839f426f3f392942e7"
server_key_iv: "fbc8a5307f00448bf3be8f593e6d54a1"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.492601 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "088f7af6aa4d46b999db5840d601d33c"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "03f63b854b2692839f426f3f392942e7"
server_key_iv: "fbc8a5307f00448bf3be8f593e6d54a1"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.496671 6208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 16:08:02.499285 6231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.500519 6208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:08:02.500726 6208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "088f7af6aa4d46b999db5840d601d33c"
format_stamp: "Formatted at 2026-08-06 16:08:02 on dist-test-slave-7kzw"
server_key: "03f63b854b2692839f426f3f392942e7"
server_key_iv: "fbc8a5307f00448bf3be8f593e6d54a1"
server_key_version: "encryptionkey@0"
I20260806 16:08:02.500849 6208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:02.517938 6208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:02.521327 6208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:02.521553 6208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:02.522210 6208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:02.523138 6208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:02.523211 6208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.523288 6208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:02.523342 6208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:02.533025 6208 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:46057
I20260806 16:08:02.533103 6383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:46057 every 8 connection(s)
I20260806 16:08:02.534047 6208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:08:02.536039 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 6208
I20260806 16:08:02.536209 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:08:02.536558 13904 external_mini_cluster.cc:1428] Setting key 29dc11af610cb8a9b5684515130368cd
Aug 06 16:08:02 dist-test-slave-7kzw krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032482, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:02.549150 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.536255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:44729 (local address 127.13.148.62:46063)
0806 16:08:02.536444 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:02.536449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:02.537547 (+ 1098us) server_negotiation.cc:407] Connection header received
0806 16:08:02.539273 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:02.539276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:02.539335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:02.539418 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:02.540702 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.541244 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.542393 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.542697 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.542812 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.545428 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:02.545431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:02.545433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:02.545456 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:02.547026 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:02.547549 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:02.547552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:02.547553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:02.547643 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:02.547989 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:02.547993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:02.547994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:02.548239 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:02.548339 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:02.548868 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 16:08:02.548989 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:08:02.550060 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.536680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46063 (local address 127.13.148.3:44729)
0806 16:08:02.537375 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:02.537440 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:02.539022 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:02.539624 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:02.539637 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:02.540021 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:02.540557 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.540571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.541373 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.541376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.542261 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.542270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.542854 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.542858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.543035 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.543684 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:02.543701 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:02.545242 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:02.547142 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:02.547148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:02.547161 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:02.547450 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:02.547730 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:02.547733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:02.547734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:02.547873 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:02.548372 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:02.548378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:02.548627 (+ 249us) client_negotiation.cc:769] Sending connection context
0806 16:08:02.548799 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 16:08:02.549064 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":158,"threads_started":1}
I20260806 16:08:02.551673 6384 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46063
I20260806 16:08:02.551975 6384 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:02.552491 6384 heartbeater.cc:511] Master 127.13.148.62:46063 requested a full tablet report, sending...
I20260806 16:08:02.553716 5756 ts_manager.cc:194] Registered new tserver with Master: 088f7af6aa4d46b999db5840d601d33c (127.13.148.3:46057)
I20260806 16:08:02.554323 5756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:44729
I20260806 16:08:02.563913 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:08:02.574638 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.566135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.13.148.62:46063)
0806 16:08:02.566306 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:02.566311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:02.566573 (+ 262us) server_negotiation.cc:407] Connection header received
0806 16:08:02.566705 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:02.566707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:02.566751 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:02.566819 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:02.567630 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.568147 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.568983 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.569204 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.569289 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.571099 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:02.571102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:02.571104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:02.571128 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:02.572944 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:02.573364 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:02.573367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:02.573368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:02.573446 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:02.573696 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:02.573699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:02.573701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:02.573929 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:02.574044 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:02.574320 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 16:08:02.574418 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:08:02.578212 5756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:08:02.581461 5756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:08:02.596740 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.590908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.13.148.1:45579)
0806 16:08:02.591351 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:02.591368 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:02.591391 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:08:02.591528 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:02.591533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:02.591768 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:02.591951 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:02.592541 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.593569 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.595213 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.596050 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.596177 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.596334 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:02.596482 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 16:08:02.596579 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":90,"threads_started":1}
I20260806 16:08:02.596773 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.590779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45579 (local address 127.0.0.1:54712)
0806 16:08:02.591173 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:02.591225 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:02.591350 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:02.592193 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:02.592200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:02.592224 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:02.592404 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.592413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.593703 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.593709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.595061 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.595073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.596170 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.596174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.596286 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.596327 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 16:08:02.596438 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 16:08:02.596550 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":22,"thread_start_us":120,"threads_started":1}
I20260806 16:08:02.597625 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.591986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39503 (local address 127.0.0.1:45142)
0806 16:08:02.592210 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:02.592222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:02.592294 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:02.593532 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:02.593535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:02.593555 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:02.593672 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.593680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.594929 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.594936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.596007 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.596018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.597191 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.597194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.597306 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.597337 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 16:08:02.597402 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 16:08:02.597471 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260806 16:08:02.597797 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.591563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46057 (local address 127.0.0.1:36616)
0806 16:08:02.591908 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:02.591921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:02.592002 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:02.592704 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:02.592707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:02.592725 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:02.592849 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.592858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.594458 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.594485 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.595864 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:02.595875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.597381 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.597385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:02.597473 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.597496 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:08:02.597576 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 16:08:02.597653 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260806 16:08:02.598119 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.592506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.13.148.2:39503)
0806 16:08:02.593024 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:02.593037 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:02.593060 (+ 23us) server_negotiation.cc:407] Connection header received
0806 16:08:02.593134 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:02.593141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:02.593384 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:02.593533 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:02.593787 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.594802 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.596152 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.596979 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.597101 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.597237 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:02.597852 (+ 615us) server_negotiation.cc:299] Negotiation successful
0806 16:08:02.597950 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":133,"threads_started":1}
I20260806 16:08:02.598656 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:02.591777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.13.148.3:46057)
0806 16:08:02.592244 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:02.592261 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:02.592283 (+ 22us) server_negotiation.cc:407] Connection header received
0806 16:08:02.592358 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:02.592363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:02.592543 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:02.593004 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:02.593168 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.594258 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.596006 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:02.597218 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:02.597357 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:02.597502 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:02.597605 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 16:08:02.598506 (+ 901us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":150,"threads_started":1}
I20260806 16:08:02.600339 5930 tablet_service.cc:1467] Processing CreateTablet for tablet ba5dd1b10d7a4767a56cc59d95ac29e1 (DEFAULT_TABLE table=test-table [id=ceaf787872754c4e87f867574b07c4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:02.601016 6298 tablet_service.cc:1467] Processing CreateTablet for tablet ba5dd1b10d7a4767a56cc59d95ac29e1 (DEFAULT_TABLE table=test-table [id=ceaf787872754c4e87f867574b07c4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:02.600939 6111 tablet_service.cc:1467] Processing CreateTablet for tablet ba5dd1b10d7a4767a56cc59d95ac29e1 (DEFAULT_TABLE table=test-table [id=ceaf787872754c4e87f867574b07c4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:02.601464 5930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5dd1b10d7a4767a56cc59d95ac29e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:02.601969 6111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5dd1b10d7a4767a56cc59d95ac29e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:02.601975 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5dd1b10d7a4767a56cc59d95ac29e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:02.608831 6406 tablet_bootstrap.cc:492] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa: Bootstrap starting.
I20260806 16:08:02.609863 6407 tablet_bootstrap.cc:492] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Bootstrap starting.
I20260806 16:08:02.610041 6408 tablet_bootstrap.cc:492] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Bootstrap starting.
I20260806 16:08:02.611109 6406 tablet_bootstrap.cc:654] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:02.611857 6406 log.cc:824] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:02.612036 6407 tablet_bootstrap.cc:654] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:02.612036 6408 tablet_bootstrap.cc:654] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:02.613440 6407 log.cc:824] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:02.613440 6408 log.cc:824] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:02.613946 6406 tablet_bootstrap.cc:492] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa: No bootstrap required, opened a new log
I20260806 16:08:02.614169 6406 ts_tablet_manager.cc:1397] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:08:02.616195 6407 tablet_bootstrap.cc:492] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: No bootstrap required, opened a new log
I20260806 16:08:02.616421 6407 ts_tablet_manager.cc:1397] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 16:08:02.618897 6406 raft_consensus.cc:348] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:02.619199 6406 raft_consensus.cc:374] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:02.619259 6406 raft_consensus.cc:729] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e33c5478e94cac965ed665d3abeaaa, State: Initialized, Role: FOLLOWER
I20260806 16:08:02.619665 6408 tablet_bootstrap.cc:492] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: No bootstrap required, opened a new log
I20260806 16:08:02.619724 6406 consensus_queue.cc:260] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:02.619900 6408 ts_tablet_manager.cc:1397] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260806 16:08:02.620427 6015 heartbeater.cc:503] Master 127.13.148.62:46063 was elected leader, sending a full tablet report...
I20260806 16:08:02.620275 6407 raft_consensus.cc:348] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:02.620518 6407 raft_consensus.cc:374] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:02.620651 6407 raft_consensus.cc:729] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088f7af6aa4d46b999db5840d601d33c, State: Initialized, Role: FOLLOWER
I20260806 16:08:02.620820 6406 ts_tablet_manager.cc:1428] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 16:08:02.621076 6407 consensus_queue.cc:260] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:02.622092 6407 ts_tablet_manager.cc:1428] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 16:08:02.622453 6384 heartbeater.cc:503] Master 127.13.148.62:46063 was elected leader, sending a full tablet report...
I20260806 16:08:02.632625 6408 raft_consensus.cc:348] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:02.632896 6408 raft_consensus.cc:374] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:02.632959 6408 raft_consensus.cc:729] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e70f553ab24fb9a6bb37a0c58b302d, State: Initialized, Role: FOLLOWER
I20260806 16:08:02.633437 6408 consensus_queue.cc:260] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:02.634166 6200 heartbeater.cc:503] Master 127.13.148.62:46063 was elected leader, sending a full tablet report...
I20260806 16:08:02.634478 6408 ts_tablet_manager.cc:1428] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
W20260806 16:08:02.785164 6385 tablet.cc:2369] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:08:02.801832 6201 tablet.cc:2369] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:08:02.828557 6016 tablet.cc:2369] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:08:03.080499 6414 raft_consensus.cc:482] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:08:03.080711 6414 raft_consensus.cc:504] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:03.081918 6414 leader_election.cc:288] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 088f7af6aa4d46b999db5840d601d33c (127.13.148.3:46057), f7e70f553ab24fb9a6bb37a0c58b302d (127.13.148.2:39503)
I20260806 16:08:03.085014 6416 raft_consensus.cc:482] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:08:03.085256 6416 raft_consensus.cc:504] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:03.086621 6416 leader_election.cc:288] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e33c5478e94cac965ed665d3abeaaa (127.13.148.1:45579), f7e70f553ab24fb9a6bb37a0c58b302d (127.13.148.2:39503)
I20260806 16:08:03.088363 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.082385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:36061 (local address 127.13.148.3:46057)
0806 16:08:03.082550 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:03.082556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:03.082849 (+ 293us) server_negotiation.cc:407] Connection header received
0806 16:08:03.083159 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:03.083175 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:03.083243 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:03.083297 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:03.083858 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.084779 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.086677 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.087393 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.087510 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.087561 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:03.088142 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 16:08:03.088231 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:08:03.088409 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.082329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46057 (local address 127.13.148.1:36061)
0806 16:08:03.082740 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:03.082762 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:03.082944 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:03.083496 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:03.083500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:03.083549 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:03.083685 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.083697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.084969 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.084973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.086502 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.086517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.087730 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.087733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.087883 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.088013 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 16:08:03.088118 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 16:08:03.088201 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":94,"thread_start_us":200,"threads_started":1}
I20260806 16:08:03.089059 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5dd1b10d7a4767a56cc59d95ac29e1" candidate_uuid: "c1e33c5478e94cac965ed665d3abeaaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088f7af6aa4d46b999db5840d601d33c" is_pre_election: true
I20260806 16:08:03.089296 6323 raft_consensus.cc:2454] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e33c5478e94cac965ed665d3abeaaa in term 0.
I20260806 16:08:03.089352 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.082980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:33063 (local address 127.13.148.2:39503)
0806 16:08:03.083098 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:03.083102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:03.083397 (+ 295us) server_negotiation.cc:407] Connection header received
0806 16:08:03.083498 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:03.083501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:03.083611 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:03.083708 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:03.083960 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.084788 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.086228 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.086797 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.086926 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.087004 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:03.089112 (+ 2108us) server_negotiation.cc:299] Negotiation successful
0806 16:08:03.089211 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:08:03.089932 5867 leader_election.cc:302] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088f7af6aa4d46b999db5840d601d33c, c1e33c5478e94cac965ed665d3abeaaa; no voters:
I20260806 16:08:03.089984 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.082897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39503 (local address 127.13.148.1:33063)
0806 16:08:03.083270 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:03.083283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:03.083398 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:03.083717 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:03.083720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:03.083736 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:03.083831 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.083837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.084897 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.084907 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.086076 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.086086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.087831 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.087838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.088383 (+ 545us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.088763 (+ 380us) client_negotiation.cc:769] Sending connection context
0806 16:08:03.089720 (+ 957us) client_negotiation.cc:240] Negotiation successful
0806 16:08:03.089806 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":355,"thread_start_us":122,"threads_started":1}
I20260806 16:08:03.090262 6414 raft_consensus.cc:2781] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:08:03.090355 6414 raft_consensus.cc:482] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:08:03.090428 6414 raft_consensus.cc:3037] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:03.091797 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.087054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39503 (local address 127.13.148.3:41203)
0806 16:08:03.087459 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:03.087476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:03.087588 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:03.087948 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:03.087951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:03.087989 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:03.088131 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.088137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.089084 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.089090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.090317 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.090330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.091274 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.091278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.091416 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.091447 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 16:08:03.091544 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 16:08:03.091621 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":81,"threads_started":1}
I20260806 16:08:03.091867 6134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5dd1b10d7a4767a56cc59d95ac29e1" candidate_uuid: "c1e33c5478e94cac965ed665d3abeaaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" is_pre_election: true
I20260806 16:08:03.091872 6414 raft_consensus.cc:504] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:03.092149 6134 raft_consensus.cc:2454] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e33c5478e94cac965ed665d3abeaaa in term 0.
I20260806 16:08:03.092258 6414 leader_election.cc:288] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [CANDIDATE]: Term 1 election: Requested vote from peers 088f7af6aa4d46b999db5840d601d33c (127.13.148.3:46057), f7e70f553ab24fb9a6bb37a0c58b302d (127.13.148.2:39503)
I20260806 16:08:03.092479 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.087358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:41203 (local address 127.13.148.2:39503)
0806 16:08:03.087669 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:03.087675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:03.087690 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:08:03.087730 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:03.087733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:03.087787 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:03.087865 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:03.088256 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.088952 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.090495 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.091121 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.091248 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.091296 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:03.092103 (+ 807us) server_negotiation.cc:299] Negotiation successful
0806 16:08:03.092201 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":89,"threads_started":1}
I20260806 16:08:03.092751 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5dd1b10d7a4767a56cc59d95ac29e1" candidate_uuid: "c1e33c5478e94cac965ed665d3abeaaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088f7af6aa4d46b999db5840d601d33c"
I20260806 16:08:03.092890 6323 raft_consensus.cc:3037] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:03.092988 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.087230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:37175 (local address 127.13.148.1:45579)
0806 16:08:03.087665 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:03.087670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:03.087686 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:08:03.087731 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:03.087734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:03.087778 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:03.087870 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:03.089220 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.089961 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.091432 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.092112 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.092218 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.092282 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:03.092716 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 16:08:03.092803 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":356}
I20260806 16:08:03.093139 6134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5dd1b10d7a4767a56cc59d95ac29e1" candidate_uuid: "c1e33c5478e94cac965ed665d3abeaaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d"
I20260806 16:08:03.093273 6134 raft_consensus.cc:3037] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:03.093438 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.086722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45579 (local address 127.13.148.3:37175)
0806 16:08:03.087037 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:03.087054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:03.087154 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:03.087910 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:03.087916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:03.087937 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:03.088905 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.088922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.090154 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.090161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.091258 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.091270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.092398 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.092402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.092523 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.092561 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 16:08:03.093152 (+ 591us) client_negotiation.cc:240] Negotiation successful
0806 16:08:03.093245 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":784,"spinlock_wait_cycles":22784,"thread_start_us":65,"threads_started":1}
I20260806 16:08:03.094122 6133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5dd1b10d7a4767a56cc59d95ac29e1" candidate_uuid: "088f7af6aa4d46b999db5840d601d33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" is_pre_election: true
I20260806 16:08:03.094269 6323 raft_consensus.cc:2454] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e33c5478e94cac965ed665d3abeaaa in term 1.
I20260806 16:08:03.094275 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5dd1b10d7a4767a56cc59d95ac29e1" candidate_uuid: "088f7af6aa4d46b999db5840d601d33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e33c5478e94cac965ed665d3abeaaa" is_pre_election: true
I20260806 16:08:03.094563 5958 raft_consensus.cc:2379] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088f7af6aa4d46b999db5840d601d33c in current term 1: Already voted for candidate c1e33c5478e94cac965ed665d3abeaaa in this term.
I20260806 16:08:03.094640 5867 leader_election.cc:302] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088f7af6aa4d46b999db5840d601d33c, c1e33c5478e94cac965ed665d3abeaaa; no voters:
I20260806 16:08:03.094900 6134 raft_consensus.cc:2454] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e33c5478e94cac965ed665d3abeaaa in term 1.
I20260806 16:08:03.094993 6414 raft_consensus.cc:2781] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:03.095116 6236 leader_election.cc:302] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088f7af6aa4d46b999db5840d601d33c; no voters: c1e33c5478e94cac965ed665d3abeaaa, f7e70f553ab24fb9a6bb37a0c58b302d
I20260806 16:08:03.095280 6414 raft_consensus.cc:686] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [term 1 LEADER]: Becoming Leader. State: Replica: c1e33c5478e94cac965ed665d3abeaaa, State: Running, Role: LEADER
I20260806 16:08:03.095398 6416 raft_consensus.cc:2726] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 16:08:03.095682 6414 consensus_queue.cc:237] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } }
I20260806 16:08:03.099287 5756 catalog_manager.cc:5495] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa reported cstate change: term changed from 0 to 1, leader changed from <none> to c1e33c5478e94cac965ed665d3abeaaa (127.13.148.1). New cstate: current_term: 1 leader_uuid: "c1e33c5478e94cac965ed665d3abeaaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e33c5478e94cac965ed665d3abeaaa" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 } health_report { overall_health: UNKNOWN } } }
I20260806 16:08:03.163195 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.158972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.13.148.1:45579)
0806 16:08:03.159121 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:03.159125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:03.159397 (+ 272us) server_negotiation.cc:407] Connection header received
0806 16:08:03.159557 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:03.159560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:03.159623 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:03.159721 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:03.160025 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.160530 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.161456 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.161730 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.161835 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.162215 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:03.162746 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:03.162851 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:03.163005 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 16:08:03.163079 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:08:03.169624 6134 raft_consensus.cc:1261] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 1 FOLLOWER]: Refusing update from remote peer c1e33c5478e94cac965ed665d3abeaaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:03.169667 6323 raft_consensus.cc:1261] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 1 FOLLOWER]: Refusing update from remote peer c1e33c5478e94cac965ed665d3abeaaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:03.170342 6425 consensus_queue.cc:1035] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7e70f553ab24fb9a6bb37a0c58b302d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:03.170634 6414 consensus_queue.cc:1035] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "088f7af6aa4d46b999db5840d601d33c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:03.187757 5753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:03.187963 5753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:03.190565 5753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 165ed314f8d44df9a19e498e65c1f0d4: Sending DeleteTablet for 3 replicas of tablet ba5dd1b10d7a4767a56cc59d95ac29e1
I20260806 16:08:03.191509 5930 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5dd1b10d7a4767a56cc59d95ac29e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:54712
I20260806 16:08:03.191516 6298 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5dd1b10d7a4767a56cc59d95ac29e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36616
I20260806 16:08:03.192039 6449 tablet_replica.cc:330] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: stopping tablet replica
I20260806 16:08:03.192003 6111 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5dd1b10d7a4767a56cc59d95ac29e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:45142
I20260806 16:08:03.192332 6448 tablet_replica.cc:330] T ba5dd1b10d7a4767a56cc59d95ac29e1 P c1e33c5478e94cac965ed665d3abeaaa: stopping tablet replica
I20260806 16:08:03.192530 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 5840
I20260806 16:08:03.192579 6451 tablet_replica.cc:330] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: stopping tablet replica
I20260806 16:08:03.192884 6451 raft_consensus.cc:2229] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:03.192885 6449 raft_consensus.cc:2229] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:03.193253 6451 raft_consensus.cc:2258] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:03.193416 6449 raft_consensus.cc:2258] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:03.194607 6451 ts_tablet_manager.cc:1905] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:03.196512 6449 ts_tablet_manager.cc:1905] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:03.197717 6451 ts_tablet_manager.cc:1918] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:03.197829 6451 log.cc:1196] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/ba5dd1b10d7a4767a56cc59d95ac29e1
I20260806 16:08:03.198128 6451 ts_tablet_manager.cc:1939] T ba5dd1b10d7a4767a56cc59d95ac29e1 P f7e70f553ab24fb9a6bb37a0c58b302d: Deleting consensus metadata
I20260806 16:08:03.199443 5736 catalog_manager.cc:4841] TS f7e70f553ab24fb9a6bb37a0c58b302d (127.13.148.2:39503): tablet ba5dd1b10d7a4767a56cc59d95ac29e1 (table test-table [id=ceaf787872754c4e87f867574b07c4ea]) successfully deleted
I20260806 16:08:03.200120 6449 ts_tablet_manager.cc:1918] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:03.200246 6449 log.cc:1196] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/ba5dd1b10d7a4767a56cc59d95ac29e1
I20260806 16:08:03.200592 6449 ts_tablet_manager.cc:1939] T ba5dd1b10d7a4767a56cc59d95ac29e1 P 088f7af6aa4d46b999db5840d601d33c: Deleting consensus metadata
I20260806 16:08:03.201722 5736 catalog_manager.cc:4841] TS 088f7af6aa4d46b999db5840d601d33c (127.13.148.3:46057): tablet ba5dd1b10d7a4767a56cc59d95ac29e1 (table test-table [id=ceaf787872754c4e87f867574b07c4ea]) successfully deleted
W20260806 16:08:03.203549 5736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:45579 (error 108)
I20260806 16:08:03.203651 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 6020
I20260806 16:08:03.209769 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.209308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45579 (local address 127.0.0.1:54724)
0806 16:08:03.209594 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:03.209666 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":128,"threads_started":1}
W20260806 16:08:03.210001 5736 catalog_manager.cc:4584] TS c1e33c5478e94cac965ed665d3abeaaa (127.13.148.1:45579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba5dd1b10d7a4767a56cc59d95ac29e1: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45579: connect: Connection refused (error 111)
I20260806 16:08:03.213871 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 6208
I20260806 16:08:03.224061 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 5705
2026-08-06T16:08:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3093 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T16:08:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:03Z Disabled control of system clock
I20260806 16:08:03.259357 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40661
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:08:03.375649 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:03.375913 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:03.375950 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:03.379984 6457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:03.380048 6457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:03.380064 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:03.380079 6457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:03.380095 6457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:03.384932 6457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:03.386039 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:03.387497 6457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:03.396109 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:03.396200 6463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:03.396428 6457 server_base.cc:1034] running on GCE node
W20260806 16:08:03.396134 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:03.397033 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:03.398108 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:03.399319 6457 hybrid_clock.cc:648] HybridClock initialized: now 1786032483399292 us; error 31 us; skew 500 ppm
I20260806 16:08:03.401250 6457 webserver.cc:460] Webserver started at http://127.13.148.62:32841/ using document root <none> and password file <none>
I20260806 16:08:03.401823 6457 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:03.401914 6457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:03.402156 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:03.404196 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "8d647b373ca44d8bb3c7b20f1fd0d963"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "35a9a08ec4cd6e92c79be064a703fe56"
server_key_iv: "443d0a6481159450fb80229182058ae7"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.404753 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "8d647b373ca44d8bb3c7b20f1fd0d963"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "35a9a08ec4cd6e92c79be064a703fe56"
server_key_iv: "443d0a6481159450fb80229182058ae7"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.408526 6457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:08:03.411020 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:03.412166 6457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:03.412308 6457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "8d647b373ca44d8bb3c7b20f1fd0d963"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "35a9a08ec4cd6e92c79be064a703fe56"
server_key_iv: "443d0a6481159450fb80229182058ae7"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.412420 6457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:03.438452 6457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:03.439258 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:03.439462 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:03.446517 6457 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40661
I20260806 16:08:03.446558 6549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40661 every 8 connection(s)
I20260806 16:08:03.447546 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:03.450776 6550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:03.455940 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 6457
I20260806 16:08:03.456075 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:03.456353 13904 external_mini_cluster.cc:1428] Setting key 1f838aa4eee744b8edb1ca4e8d29d47c
I20260806 16:08:03.456699 6550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963: Bootstrap starting.
I20260806 16:08:03.459141 6550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:03.459937 6550 log.cc:824] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:03.462030 6550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963: No bootstrap required, opened a new log
I20260806 16:08:03.466722 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.457810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.13.148.62:40661)
0806 16:08:03.458436 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:03.458450 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:03.458536 (+ 86us) server_negotiation.cc:407] Connection header received
0806 16:08:03.459151 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:03.459168 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:03.459629 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:03.459910 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:03.460297 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.461244 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.462393 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.462908 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.463029 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.463912 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:03.463917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:03.463929 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:03.464358 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:03.464421 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:03.464442 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:03.464522 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:03.464859 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 16:08:03.465213 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
I20260806 16:08:03.467695 6550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d647b373ca44d8bb3c7b20f1fd0d963" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40661 } }
I20260806 16:08:03.468006 6550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:03.468075 6550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d647b373ca44d8bb3c7b20f1fd0d963, State: Initialized, Role: FOLLOWER
I20260806 16:08:03.468595 6550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d647b373ca44d8bb3c7b20f1fd0d963" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40661 } }
I20260806 16:08:03.468766 6550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:03.468839 6550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:03.468940 6550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:03.470191 6550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d647b373ca44d8bb3c7b20f1fd0d963" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40661 } }
I20260806 16:08:03.470623 6550 leader_election.cc:302] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d647b373ca44d8bb3c7b20f1fd0d963; no voters:
I20260806 16:08:03.470986 6550 leader_election.cc:288] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:03.471143 6556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:03.471385 6556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [term 1 LEADER]: Becoming Leader. State: Replica: 8d647b373ca44d8bb3c7b20f1fd0d963, State: Running, Role: LEADER
I20260806 16:08:03.471765 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d647b373ca44d8bb3c7b20f1fd0d963" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40661 } }
I20260806 16:08:03.472378 6550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:03.473099 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d647b373ca44d8bb3c7b20f1fd0d963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d647b373ca44d8bb3c7b20f1fd0d963" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40661 } } }
I20260806 16:08:03.473258 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:03.474841 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d647b373ca44d8bb3c7b20f1fd0d963. Latest consensus state: current_term: 1 leader_uuid: "8d647b373ca44d8bb3c7b20f1fd0d963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d647b373ca44d8bb3c7b20f1fd0d963" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40661 } } }
I20260806 16:08:03.475021 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963 [sys.catalog]: This master's current role is: LEADER
W20260806 16:08:03.480042 6575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:03.480154 6575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:03.480270 6562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:03.481448 6562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:03.490677 6562 catalog_manager.cc:1347] Generated new cluster ID: b5ec8f1fbdbb48609e58a0cf2bddab3a
I20260806 16:08:03.490773 6562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:03.505151 6562 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:03.506403 6562 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:03.519583 6562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963: Generated new TSK 0
I20260806 16:08:03.520457 6562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:03.528513 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40661
--builtin_ntp_servers=127.13.148.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:08:03.644867 6582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:03.645252 6582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:03.645337 6582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:03.649492 6582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:03.649561 6582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:03.649648 6582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:08:03.654320 6582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:03.655503 6582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:03.656960 6582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:03.668313 6592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:03.668474 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:03.668628 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:03.669371 6582 server_base.cc:1034] running on GCE node
I20260806 16:08:03.669801 6582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:03.670415 6582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:03.671679 6582 hybrid_clock.cc:648] HybridClock initialized: now 1786032483671660 us; error 38 us; skew 500 ppm
I20260806 16:08:03.673880 6582 webserver.cc:460] Webserver started at http://127.13.148.1:45599/ using document root <none> and password file <none>
I20260806 16:08:03.674544 6582 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:03.674640 6582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:03.674881 6582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:03.677275 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "abd8542da9e2448cb0303bbdd8644171"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "fefc3327908efaff255918212141dcae"
server_key_iv: "634b61ba8d444fd525c7a0aeed184d01"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.677865 6582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "abd8542da9e2448cb0303bbdd8644171"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "fefc3327908efaff255918212141dcae"
server_key_iv: "634b61ba8d444fd525c7a0aeed184d01"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.681672 6582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 16:08:03.684300 6600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:03.685600 6582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:03.685772 6582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "abd8542da9e2448cb0303bbdd8644171"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "fefc3327908efaff255918212141dcae"
server_key_iv: "634b61ba8d444fd525c7a0aeed184d01"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.685897 6582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:03.703658 6582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:03.704380 6582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:03.704592 6582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:03.705248 6582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:03.706297 6582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:03.706367 6582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:03.706439 6582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:03.706508 6582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 16:08:03.716286 6582 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:33081
I20260806 16:08:03.716353 6763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:33081 every 8 connection(s)
I20260806 16:08:03.717315 6582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:08:03.726729 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 6582
I20260806 16:08:03.726884 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:08:03.726905 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.719314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:52789 (local address 127.13.148.62:40661)
0806 16:08:03.719459 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:03.719463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:03.720546 (+ 1083us) server_negotiation.cc:407] Connection header received
0806 16:08:03.722049 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:03.722052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:03.722112 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:03.722221 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:03.723174 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.723737 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.724882 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.725112 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.725243 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.726386 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:03.726390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:03.726392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:03.726403 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:03.726423 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:03.726432 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:03.726529 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:03.726688 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 16:08:03.726753 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:08:03.727240 13904 external_mini_cluster.cc:1428] Setting key d4d6190dbaa4d0d50f73320b0b6bf684
I20260806 16:08:03.727936 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.719735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40661 (local address 127.13.148.1:52789)
0806 16:08:03.720397 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:03.720438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:03.721828 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:03.722397 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:03.722405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:03.722544 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:03.723032 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.723046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.723877 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.723880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.724745 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.724755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.725299 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.725304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.725462 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.726102 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:03.726103 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:03.726144 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:03.726213 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:03.726531 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:03.726557 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:03.726563 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 16:08:03.726728 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 16:08:03.727008 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260806 16:08:03.729228 6764 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40661
I20260806 16:08:03.729482 6764 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:03.729605 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40661
--builtin_ntp_servers=127.13.148.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:08:03.730072 6764 heartbeater.cc:511] Master 127.13.148.62:40661 requested a full tablet report, sending...
I20260806 16:08:03.731813 6495 ts_manager.cc:194] Registered new tserver with Master: abd8542da9e2448cb0303bbdd8644171 (127.13.148.1:33081)
I20260806 16:08:03.733248 6495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.1:52789
W20260806 16:08:03.846983 6771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:03.847251 6771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:03.847306 6771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:03.851274 6771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:03.851330 6771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:03.851385 6771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:08:03.855985 6771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:03.856966 6771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:03.858289 6771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:03.867935 6781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:03.867923 6779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:03.867920 6778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:03.868075 6771 server_base.cc:1034] running on GCE node
I20260806 16:08:03.868712 6771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:03.869318 6771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:03.870671 6771 hybrid_clock.cc:648] HybridClock initialized: now 1786032483870652 us; error 29 us; skew 500 ppm
I20260806 16:08:03.872593 6771 webserver.cc:460] Webserver started at http://127.13.148.2:38777/ using document root <none> and password file <none>
I20260806 16:08:03.873219 6771 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:03.873289 6771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:03.873564 6771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:03.875789 6771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "9bfcf0c441cf4270b732abd76c320054"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "48ddd47b4846bb033c216fc50fe3bc2b"
server_key_iv: "3b6c2e909facf02150e8a898ebdae37f"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.876384 6771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "9bfcf0c441cf4270b732abd76c320054"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "48ddd47b4846bb033c216fc50fe3bc2b"
server_key_iv: "3b6c2e909facf02150e8a898ebdae37f"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.880280 6771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:08:03.882748 6787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:03.884089 6771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:03.884253 6771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "9bfcf0c441cf4270b732abd76c320054"
format_stamp: "Formatted at 2026-08-06 16:08:03 on dist-test-slave-7kzw"
server_key: "48ddd47b4846bb033c216fc50fe3bc2b"
server_key_iv: "3b6c2e909facf02150e8a898ebdae37f"
server_key_version: "encryptionkey@0"
I20260806 16:08:03.884382 6771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:03.900813 6771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:03.901536 6771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:03.901757 6771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:03.902386 6771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:03.903514 6771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:03.903590 6771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:03.903681 6771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:03.903736 6771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:03.913674 6771 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:43791
I20260806 16:08:03.913688 6951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:43791 every 8 connection(s)
I20260806 16:08:03.914749 6771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:08:03.918754 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 6771
I20260806 16:08:03.918918 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:08:03.919236 13904 external_mini_cluster.cc:1428] Setting key 62f7fe51626c9129160b45ef25c99601
I20260806 16:08:03.921471 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40661
--builtin_ntp_servers=127.13.148.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:08:03.925354 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.916690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:54973 (local address 127.13.148.62:40661)
0806 16:08:03.916865 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:03.916870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:03.918100 (+ 1230us) server_negotiation.cc:407] Connection header received
0806 16:08:03.920084 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:03.920087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:03.920138 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:03.920264 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:03.921333 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.921990 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.923234 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.923458 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.923585 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.924828 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:03.924831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:03.924833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:03.924846 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:03.924869 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:03.924877 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:03.924947 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:03.925174 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 16:08:03.925230 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 16:08:03.926586 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:03.917196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40661 (local address 127.13.148.2:54973)
0806 16:08:03.917945 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:03.917990 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:03.919642 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:03.920455 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:03.920461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:03.920551 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:03.921172 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.921187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.922135 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.922145 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.923092 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:03.923103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:03.923636 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:03.923641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:03.923809 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:03.924494 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:03.924496 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:03.924543 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:03.924643 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:03.924972 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:03.924985 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:03.924995 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 16:08:03.925147 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 16:08:03.925453 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":127,"threads_started":1}
I20260806 16:08:03.927971 6952 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40661
I20260806 16:08:03.928238 6952 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:03.928800 6952 heartbeater.cc:511] Master 127.13.148.62:40661 requested a full tablet report, sending...
I20260806 16:08:03.929957 6495 ts_manager.cc:194] Registered new tserver with Master: 9bfcf0c441cf4270b732abd76c320054 (127.13.148.2:43791)
I20260806 16:08:03.930667 6495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.2:54973
W20260806 16:08:04.038185 6956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:04.038509 6956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:04.038591 6956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:04.042729 6956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:04.042809 6956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:04.042891 6956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:08:04.047623 6956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:04.048725 6956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:04.050145 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:04.059773 6965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:04.059785 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:04.059881 6964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:04.059940 6956 server_base.cc:1034] running on GCE node
I20260806 16:08:04.060448 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:04.061000 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:04.062172 6956 hybrid_clock.cc:648] HybridClock initialized: now 1786032484062157 us; error 25 us; skew 500 ppm
I20260806 16:08:04.064055 6956 webserver.cc:460] Webserver started at http://127.13.148.3:33525/ using document root <none> and password file <none>
I20260806 16:08:04.064672 6956 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:04.064739 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:04.064947 6956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:04.066991 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "9bf62255ee204f418b7a6a8065c137c2"
format_stamp: "Formatted at 2026-08-06 16:08:04 on dist-test-slave-7kzw"
server_key: "610bdf6fcc867c98c1120551683c63f9"
server_key_iv: "d8d6eb7df5b4634d5ae5770d9df20368"
server_key_version: "encryptionkey@0"
I20260806 16:08:04.067580 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "9bf62255ee204f418b7a6a8065c137c2"
format_stamp: "Formatted at 2026-08-06 16:08:04 on dist-test-slave-7kzw"
server_key: "610bdf6fcc867c98c1120551683c63f9"
server_key_iv: "d8d6eb7df5b4634d5ae5770d9df20368"
server_key_version: "encryptionkey@0"
I20260806 16:08:04.071215 6956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:08:04.073756 6976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:04.075099 6956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:08:04.075222 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "9bf62255ee204f418b7a6a8065c137c2"
format_stamp: "Formatted at 2026-08-06 16:08:04 on dist-test-slave-7kzw"
server_key: "610bdf6fcc867c98c1120551683c63f9"
server_key_iv: "d8d6eb7df5b4634d5ae5770d9df20368"
server_key_version: "encryptionkey@0"
I20260806 16:08:04.075325 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:04.087558 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:04.088251 6956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:04.088441 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:04.089074 6956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:04.090060 6956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:04.090108 6956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:04.090171 6956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:04.090204 6956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:04.099290 6956 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:42759
I20260806 16:08:04.099324 7139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:42759 every 8 connection(s)
I20260806 16:08:04.100201 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:08:04.109530 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.102034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:40489 (local address 127.13.148.62:40661)
0806 16:08:04.102187 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:04.102192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:04.103349 (+ 1157us) server_negotiation.cc:407] Connection header received
0806 16:08:04.104760 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:04.104763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:04.104812 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:04.104889 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:04.105776 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.106307 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.107529 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.107784 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.107871 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.109051 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:04.109054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:04.109057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:04.109069 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:04.109089 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:04.109096 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:04.109171 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:04.109345 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 16:08:04.109411 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:08:04.110023 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 6956
I20260806 16:08:04.110148 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:08:04.110446 13904 external_mini_cluster.cc:1428] Setting key 4b21f545e6ac56b2eb382f7b421649d3
I20260806 16:08:04.110718 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.102533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40661 (local address 127.13.148.3:40489)
0806 16:08:04.103204 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:04.103245 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:04.104518 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:04.105053 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:04.105060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:04.105158 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:04.105640 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.105652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.106424 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.106427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.107406 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.107426 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.107924 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.107927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.108086 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.108784 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:04.108786 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:04.108834 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:04.108897 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:04.109185 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:04.109192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:04.109197 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:08:04.109301 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:08:04.109516 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260806 16:08:04.111898 7140 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40661
I20260806 16:08:04.112151 7140 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:04.112737 7140 heartbeater.cc:511] Master 127.13.148.62:40661 requested a full tablet report, sending...
I20260806 16:08:04.113714 6497 ts_manager.cc:194] Registered new tserver with Master: 9bf62255ee204f418b7a6a8065c137c2 (127.13.148.3:42759)
I20260806 16:08:04.114312 6497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.3:40489
I20260806 16:08:04.124446 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:08:04.129980 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.126419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.13.148.62:40661)
0806 16:08:04.126554 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:04.126558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:04.126729 (+ 171us) server_negotiation.cc:407] Connection header received
0806 16:08:04.126823 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:04.126825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:04.126865 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:04.126925 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:04.127164 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.127682 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.128551 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.128772 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.128859 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.129624 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:04.129628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:04.129630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:04.129645 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:04.129665 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:04.129673 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:04.129732 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:04.129825 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 16:08:04.129873 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:08:04.132936 6497 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:08:04.135952 6497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:08:04.150905 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.144769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.13.148.3:42759)
0806 16:08:04.145471 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:04.145489 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:04.145518 (+ 29us) server_negotiation.cc:407] Connection header received
0806 16:08:04.145597 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:04.145604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:04.145823 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:04.145986 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:04.146527 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.147678 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.149644 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.150172 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.150299 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.150534 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:04.150631 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 16:08:04.150728 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":124,"threads_started":1}
I20260806 16:08:04.150951 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.144575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42759 (local address 127.0.0.1:54800)
0806 16:08:04.144986 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:04.145039 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:04.145175 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:04.146181 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:04.146201 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:04.146229 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:04.146386 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.146392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.147812 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.147816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.149471 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.149482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.150299 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.150303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.150406 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.150443 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 16:08:04.150603 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 16:08:04.150742 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":222,"spinlock_wait_cycles":2304,"thread_start_us":101,"threads_started":1}
I20260806 16:08:04.151198 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.145584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.13.148.2:43791)
0806 16:08:04.146080 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:04.146095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:04.146211 (+ 116us) server_negotiation.cc:407] Connection header received
0806 16:08:04.146335 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:04.146340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:04.146540 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:04.146713 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:04.146901 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.147711 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.149602 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.150360 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.150453 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.150632 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:04.150917 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 16:08:04.151016 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":150,"threads_started":1}
I20260806 16:08:04.151295 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.145634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.13.148.1:33081)
0806 16:08:04.146313 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:04.146324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:04.146344 (+ 20us) server_negotiation.cc:407] Connection header received
0806 16:08:04.146409 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:04.146413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:04.146581 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:04.146715 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:04.147108 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.148143 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.149424 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.150218 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.150335 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.150557 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:04.151020 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 16:08:04.151114 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":126,"threads_started":1}
I20260806 16:08:04.151643 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.145317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:43791 (local address 127.0.0.1:33008)
0806 16:08:04.146109 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:04.146124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:04.146220 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:04.146701 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:04.146704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:04.146723 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:04.146803 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.146810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.147855 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.147857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.149471 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.149482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.150683 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.150686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.150763 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.150780 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:08:04.150839 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 16:08:04.151516 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":694,"thread_start_us":89,"threads_started":1}
I20260806 16:08:04.152486 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.145061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:33081 (local address 127.0.0.1:44080)
0806 16:08:04.145958 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:04.145972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:04.146063 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:04.146860 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:04.146866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:04.146887 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:04.146976 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.146983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.148288 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.148292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.149300 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.149310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.150624 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.150628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.150732 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.150765 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 16:08:04.150839 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 16:08:04.150927 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":62,"thread_start_us":102,"threads_started":1}
I20260806 16:08:04.154289 6858 tablet_service.cc:1467] Processing CreateTablet for tablet b604212cc42942f99960a6e844c8bb0f (DEFAULT_TABLE table=test-table [id=d3d046dd71894f969c996decf7495d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:04.154407 7041 tablet_service.cc:1467] Processing CreateTablet for tablet b604212cc42942f99960a6e844c8bb0f (DEFAULT_TABLE table=test-table [id=d3d046dd71894f969c996decf7495d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:04.154963 6667 tablet_service.cc:1467] Processing CreateTablet for tablet b604212cc42942f99960a6e844c8bb0f (DEFAULT_TABLE table=test-table [id=d3d046dd71894f969c996decf7495d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:04.155439 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b604212cc42942f99960a6e844c8bb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:04.155483 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b604212cc42942f99960a6e844c8bb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:04.155864 6667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b604212cc42942f99960a6e844c8bb0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:04.161523 7160 tablet_bootstrap.cc:492] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: Bootstrap starting.
I20260806 16:08:04.162938 7161 tablet_bootstrap.cc:492] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171: Bootstrap starting.
I20260806 16:08:04.164304 7160 tablet_bootstrap.cc:654] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:04.164999 7161 tablet_bootstrap.cc:654] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:04.165197 7160 log.cc:824] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:04.165829 7161 log.cc:824] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:04.165963 7162 tablet_bootstrap.cc:492] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Bootstrap starting.
I20260806 16:08:04.167289 7160 tablet_bootstrap.cc:492] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: No bootstrap required, opened a new log
I20260806 16:08:04.167528 7160 ts_tablet_manager.cc:1397] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 16:08:04.167932 7161 tablet_bootstrap.cc:492] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171: No bootstrap required, opened a new log
I20260806 16:08:04.168005 7162 tablet_bootstrap.cc:654] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:04.168129 7161 ts_tablet_manager.cc:1397] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:08:04.168844 7162 log.cc:824] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:04.170746 7162 tablet_bootstrap.cc:492] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: No bootstrap required, opened a new log
I20260806 16:08:04.170923 7162 ts_tablet_manager.cc:1397] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:08:04.171406 7160 raft_consensus.cc:348] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.171648 7160 raft_consensus.cc:374] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:04.171702 7160 raft_consensus.cc:729] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfcf0c441cf4270b732abd76c320054, State: Initialized, Role: FOLLOWER
I20260806 16:08:04.172173 7160 consensus_queue.cc:260] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.172459 7161 raft_consensus.cc:348] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.172925 6952 heartbeater.cc:503] Master 127.13.148.62:40661 was elected leader, sending a full tablet report...
I20260806 16:08:04.173187 7160 ts_tablet_manager.cc:1428] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:08:04.173934 7161 raft_consensus.cc:374] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:04.174036 7161 raft_consensus.cc:729] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd8542da9e2448cb0303bbdd8644171, State: Initialized, Role: FOLLOWER
I20260806 16:08:04.174525 7161 consensus_queue.cc:260] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.174960 7162 raft_consensus.cc:348] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.175238 6764 heartbeater.cc:503] Master 127.13.148.62:40661 was elected leader, sending a full tablet report...
I20260806 16:08:04.175271 7162 raft_consensus.cc:374] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:04.175329 7162 raft_consensus.cc:729] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf62255ee204f418b7a6a8065c137c2, State: Initialized, Role: FOLLOWER
I20260806 16:08:04.175746 7161 ts_tablet_manager.cc:1428] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 16:08:04.175876 7162 consensus_queue.cc:260] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.176661 7140 heartbeater.cc:503] Master 127.13.148.62:40661 was elected leader, sending a full tablet report...
I20260806 16:08:04.177799 7162 ts_tablet_manager.cc:1428] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260806 16:08:04.218612 6767 tablet.cc:2369] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:08:04.333312 7167 raft_consensus.cc:482] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:08:04.333590 7167 raft_consensus.cc:504] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.335023 7167 leader_election.cc:288] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf62255ee204f418b7a6a8065c137c2 (127.13.148.3:42759), 9bfcf0c441cf4270b732abd76c320054 (127.13.148.2:43791)
I20260806 16:08:04.339331 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.335817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42175 (local address 127.13.148.2:43791)
0806 16:08:04.335980 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:04.335985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:04.336026 (+ 41us) server_negotiation.cc:407] Connection header received
0806 16:08:04.336164 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:04.336167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:04.336215 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:04.336294 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:04.336619 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.337153 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.338255 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.338713 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.338811 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.338893 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:04.339110 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 16:08:04.339195 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:08:04.339370 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.335386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:50209 (local address 127.13.148.3:42759)
0806 16:08:04.335527 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:04.335530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:04.335645 (+ 115us) server_negotiation.cc:407] Connection header received
0806 16:08:04.335832 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:04.335835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:04.335909 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:04.335963 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:04.336425 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.337009 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.338307 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.338751 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.338890 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.338943 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:04.339166 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 16:08:04.339249 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:08:04.339358 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.335652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:43791 (local address 127.13.148.1:42175)
0806 16:08:04.335955 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:04.335968 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:04.336055 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:04.336313 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:04.336315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:04.336329 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:04.336478 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.336484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.337274 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.337277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.338124 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.338133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.338846 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.338851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.338943 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.339004 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 16:08:04.339065 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 16:08:04.339154 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":59,"thread_start_us":102,"threads_started":1}
I20260806 16:08:04.339365 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.335244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42759 (local address 127.13.148.1:50209)
0806 16:08:04.335541 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:04.335558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:04.335676 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:04.336147 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:04.336150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:04.336171 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:04.336301 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.336309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.337142 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.337145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.338128 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:04.338139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.338923 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.338927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:04.339016 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.339033 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:08:04.339129 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 16:08:04.339197 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":41,"thread_start_us":107,"threads_started":1}
I20260806 16:08:04.340132 7075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b604212cc42942f99960a6e844c8bb0f" candidate_uuid: "abd8542da9e2448cb0303bbdd8644171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf62255ee204f418b7a6a8065c137c2" is_pre_election: true
I20260806 16:08:04.340134 6887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b604212cc42942f99960a6e844c8bb0f" candidate_uuid: "abd8542da9e2448cb0303bbdd8644171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfcf0c441cf4270b732abd76c320054" is_pre_election: true
I20260806 16:08:04.340435 6887 raft_consensus.cc:2454] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd8542da9e2448cb0303bbdd8644171 in term 0.
I20260806 16:08:04.340440 7075 raft_consensus.cc:2454] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd8542da9e2448cb0303bbdd8644171 in term 0.
I20260806 16:08:04.340946 6606 leader_election.cc:302] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfcf0c441cf4270b732abd76c320054, abd8542da9e2448cb0303bbdd8644171; no voters:
I20260806 16:08:04.341254 7167 raft_consensus.cc:2781] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:08:04.341336 7167 raft_consensus.cc:482] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:08:04.341368 7167 raft_consensus.cc:3037] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:04.342440 7167 raft_consensus.cc:504] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.342813 7167 leader_election.cc:288] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf62255ee204f418b7a6a8065c137c2 (127.13.148.3:42759), 9bfcf0c441cf4270b732abd76c320054 (127.13.148.2:43791)
I20260806 16:08:04.343302 7075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b604212cc42942f99960a6e844c8bb0f" candidate_uuid: "abd8542da9e2448cb0303bbdd8644171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf62255ee204f418b7a6a8065c137c2"
I20260806 16:08:04.343420 6887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b604212cc42942f99960a6e844c8bb0f" candidate_uuid: "abd8542da9e2448cb0303bbdd8644171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfcf0c441cf4270b732abd76c320054"
I20260806 16:08:04.343439 7075 raft_consensus.cc:3037] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:04.343572 6887 raft_consensus.cc:3037] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:04.344632 7075 raft_consensus.cc:2454] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd8542da9e2448cb0303bbdd8644171 in term 1.
I20260806 16:08:04.344612 6887 raft_consensus.cc:2454] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd8542da9e2448cb0303bbdd8644171 in term 1.
I20260806 16:08:04.344964 6606 leader_election.cc:302] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bf62255ee204f418b7a6a8065c137c2, abd8542da9e2448cb0303bbdd8644171; no voters:
I20260806 16:08:04.345191 7167 raft_consensus.cc:2781] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:04.345438 7167 raft_consensus.cc:686] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [term 1 LEADER]: Becoming Leader. State: Replica: abd8542da9e2448cb0303bbdd8644171, State: Running, Role: LEADER
I20260806 16:08:04.345815 7167 consensus_queue.cc:237] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } }
I20260806 16:08:04.349067 6495 catalog_manager.cc:5495] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 reported cstate change: term changed from 0 to 1, leader changed from <none> to abd8542da9e2448cb0303bbdd8644171 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "abd8542da9e2448cb0303bbdd8644171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abd8542da9e2448cb0303bbdd8644171" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33081 } health_report { overall_health: HEALTHY } } }
W20260806 16:08:04.351128 7142 tablet.cc:2369] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:08:04.390033 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.386091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.13.148.1:33081)
0806 16:08:04.386244 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:04.386248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:04.386262 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:08:04.386340 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:04.386342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:04.386397 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:04.386530 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:04.386807 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.387315 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.388291 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:04.388509 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:04.388689 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:04.389044 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:04.389535 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:04.389636 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:04.389780 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 16:08:04.389851 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 16:08:04.396309 7075 raft_consensus.cc:1261] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 1 FOLLOWER]: Refusing update from remote peer abd8542da9e2448cb0303bbdd8644171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:04.396309 6887 raft_consensus.cc:1261] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 1 FOLLOWER]: Refusing update from remote peer abd8542da9e2448cb0303bbdd8644171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:04.397116 7167 consensus_queue.cc:1035] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfcf0c441cf4270b732abd76c320054" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:04.397351 7172 consensus_queue.cc:1035] T b604212cc42942f99960a6e844c8bb0f P abd8542da9e2448cb0303bbdd8644171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf62255ee204f418b7a6a8065c137c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:04.408821 7178 mvcc.cc:205] Tried to move back new op lower bound from 7315589056079253504 to 7315589055883149312. Current Snapshot: MvccSnapshot[applied={T|T < 7315589056079253504}]
I20260806 16:08:04.413792 6497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:04.413955 6497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:04.416808 6497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d647b373ca44d8bb3c7b20f1fd0d963: Sending DeleteTablet for 3 replicas of tablet b604212cc42942f99960a6e844c8bb0f
I20260806 16:08:04.417675 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet b604212cc42942f99960a6e844c8bb0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:04 UTC) from {username='slave'} at 127.0.0.1:33008
I20260806 16:08:04.417682 7041 tablet_service.cc:1514] Processing DeleteTablet for tablet b604212cc42942f99960a6e844c8bb0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:04 UTC) from {username='slave'} at 127.0.0.1:54800
I20260806 16:08:04.418219 7190 tablet_replica.cc:330] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: stopping tablet replica
I20260806 16:08:04.418612 7190 raft_consensus.cc:2229] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:04.418574 6667 tablet_service.cc:1514] Processing DeleteTablet for tablet b604212cc42942f99960a6e844c8bb0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:04 UTC) from {username='slave'} at 127.0.0.1:44080
I20260806 16:08:04.418949 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 6582
I20260806 16:08:04.419015 7190 raft_consensus.cc:2258] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:04.419327 7193 tablet_replica.cc:330] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: stopping tablet replica
I20260806 16:08:04.419625 7193 raft_consensus.cc:2229] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:04.419867 7193 raft_consensus.cc:2258] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:04.428398 7193 ts_tablet_manager.cc:1905] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:04.431674 7190 ts_tablet_manager.cc:1905] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:04.431806 7193 ts_tablet_manager.cc:1918] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:04.431888 7193 log.cc:1196] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/b604212cc42942f99960a6e844c8bb0f
I20260806 16:08:04.432233 7193 ts_tablet_manager.cc:1939] T b604212cc42942f99960a6e844c8bb0f P 9bf62255ee204f418b7a6a8065c137c2: Deleting consensus metadata
I20260806 16:08:04.433560 6479 catalog_manager.cc:4841] TS 9bf62255ee204f418b7a6a8065c137c2 (127.13.148.3:42759): tablet b604212cc42942f99960a6e844c8bb0f (table test-table [id=d3d046dd71894f969c996decf7495d7e]) successfully deleted
I20260806 16:08:04.434726 7190 ts_tablet_manager.cc:1918] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:04.434835 7190 log.cc:1196] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/b604212cc42942f99960a6e844c8bb0f
I20260806 16:08:04.435144 7190 ts_tablet_manager.cc:1939] T b604212cc42942f99960a6e844c8bb0f P 9bfcf0c441cf4270b732abd76c320054: Deleting consensus metadata
I20260806 16:08:04.436427 6479 catalog_manager.cc:4841] TS 9bfcf0c441cf4270b732abd76c320054 (127.13.148.2:43791): tablet b604212cc42942f99960a6e844c8bb0f (table test-table [id=d3d046dd71894f969c996decf7495d7e]) successfully deleted
W20260806 16:08:04.440002 6476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:33081 (error 108)
I20260806 16:08:04.440294 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 6771
I20260806 16:08:04.447144 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:04.446707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:33081 (local address 127.0.0.1:44096)
0806 16:08:04.446936 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:04.447029 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260806 16:08:04.447381 6476 catalog_manager.cc:4584] TS abd8542da9e2448cb0303bbdd8644171 (127.13.148.1:33081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b604212cc42942f99960a6e844c8bb0f: Network error: Client connection negotiation failed: client connection to 127.13.148.1:33081: connect: Connection refused (error 111)
I20260806 16:08:04.452392 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 6956
I20260806 16:08:04.465284 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 6457
2026-08-06T16:08:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1243 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:04 dist-test-slave-7kzw krb5kdc[7203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:04 dist-test-slave-7kzw krb5kdc[7203](info): set up 2 sockets
Aug 06 16:08:04 dist-test-slave-7kzw krb5kdc[7203](info): commencing operation
krb5kdc: starting...
W20260806 16:08:05.705806 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:05 dist-test-slave-7kzw krb5kdc[7203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:05.838050 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42583
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:42583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:05.957573 7224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:05.957871 7224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:05.957944 7224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:05.961869 7224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:05.961957 7224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:05.961997 7224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:05.962052 7224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:05.962093 7224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:05.967195 7224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:42583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:05.968243 7224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:05.969292 7224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:05.978154 7230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:05.978154 7235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:05.978202 7231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:05.978438 7224 server_base.cc:1034] running on GCE node
I20260806 16:08:05.979003 7224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:05.980072 7224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:05.981256 7224 hybrid_clock.cc:648] HybridClock initialized: now 1786032485981236 us; error 32 us; skew 500 ppm
Aug 06 16:08:05 dist-test-slave-7kzw krb5kdc[7203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032485, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:05.987483 7224 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:05.988368 7224 webserver.cc:460] Webserver started at http://127.13.148.62:39975/ using document root <none> and password file <none>
I20260806 16:08:05.988936 7224 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:05.989029 7224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:05.989277 7224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:05.991392 7224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7"
format_stamp: "Formatted at 2026-08-06 16:08:05 on dist-test-slave-7kzw"
server_key: "0a3e9d439202c844a48aaaa29e7dab5e"
server_key_iv: "d8fac4b5a3eb58ed238ce75a66f54da6"
server_key_version: "encryptionkey@0"
I20260806 16:08:05.991957 7224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7"
format_stamp: "Formatted at 2026-08-06 16:08:05 on dist-test-slave-7kzw"
server_key: "0a3e9d439202c844a48aaaa29e7dab5e"
server_key_iv: "d8fac4b5a3eb58ed238ce75a66f54da6"
server_key_version: "encryptionkey@0"
I20260806 16:08:05.995710 7224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:08:05.998104 7248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:05.999429 7224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:05.999575 7224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7"
format_stamp: "Formatted at 2026-08-06 16:08:05 on dist-test-slave-7kzw"
server_key: "0a3e9d439202c844a48aaaa29e7dab5e"
server_key_iv: "d8fac4b5a3eb58ed238ce75a66f54da6"
server_key_version: "encryptionkey@0"
I20260806 16:08:05.999696 7224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:06.029899 7224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:06.033305 7224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:06.033669 7224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:06.041221 7224 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:42583
I20260806 16:08:06.041246 7318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:42583 every 8 connection(s)
I20260806 16:08:06.042238 7224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:06.045249 7319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:06.046308 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 7224
I20260806 16:08:06.046460 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:06.046785 13904 external_mini_cluster.cc:1428] Setting key 2014b769b828e26e8ea08088b4578174
I20260806 16:08:06.052143 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7: Bootstrap starting.
I20260806 16:08:06.054826 7319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:06.055584 7319 log.cc:824] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:08:06 dist-test-slave-7kzw krb5kdc[7203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:06.057641 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7: No bootstrap required, opened a new log
I20260806 16:08:06.061339 7319 raft_consensus.cc:348] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42583 } }
I20260806 16:08:06.061568 7319 raft_consensus.cc:374] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:06.061646 7319 raft_consensus.cc:729] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daedf4d2d3d24b2cb22ea10800a5c7b7, State: Initialized, Role: FOLLOWER
I20260806 16:08:06.062062 7319 consensus_queue.cc:260] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42583 } }
I20260806 16:08:06.062181 7319 raft_consensus.cc:388] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:06.062197 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.048242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.13.148.62:42583)
0806 16:08:06.048683 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.048705 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.048751 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:08:06.049575 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.049600 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.050008 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.050416 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:06.051430 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.052879 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.053738 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.054410 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.054620 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.057281 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:06.057286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:06.057299 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:06.057325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:06.059197 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.059653 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.059660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.059664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.059789 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.060077 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.060081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.060086 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.060470 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:06.060703 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.060971 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.061274 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":119,"threads_started":1}
I20260806 16:08:06.062278 7319 raft_consensus.cc:482] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:06.062391 7319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:06.063411 7319 raft_consensus.cc:504] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42583 } }
I20260806 16:08:06.063766 7319 leader_election.cc:302] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daedf4d2d3d24b2cb22ea10800a5c7b7; no voters:
I20260806 16:08:06.064085 7319 leader_election.cc:288] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:06.064241 7328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:06.064481 7328 raft_consensus.cc:686] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [term 1 LEADER]: Becoming Leader. State: Replica: daedf4d2d3d24b2cb22ea10800a5c7b7, State: Running, Role: LEADER
I20260806 16:08:06.064838 7328 consensus_queue.cc:237] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42583 } }
I20260806 16:08:06.065274 7319 sys_catalog.cc:564] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:06.065940 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daedf4d2d3d24b2cb22ea10800a5c7b7. Latest consensus state: current_term: 1 leader_uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42583 } } }
I20260806 16:08:06.066164 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:06.066421 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daedf4d2d3d24b2cb22ea10800a5c7b7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42583 } } }
I20260806 16:08:06.066573 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:06.066737 7342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:06.069866 7342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:06.075738 7342 catalog_manager.cc:1347] Generated new cluster ID: 3ea9de933c244dc6af1427678717ecfa
I20260806 16:08:06.075819 7342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:06.080967 7342 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:06.081964 7342 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:06.086267 7342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7: Generated new TSK 0
I20260806 16:08:06.086896 7342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:08:06.121999 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42583
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:06.241273 7357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:06.241573 7357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:06.241622 7357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:06.245667 7357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:06.245744 7357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:06.245805 7357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:08:06.251202 7357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:06.252287 7357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:06.253340 7357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:06.263387 7365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:06.263370 7363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:06.263478 7357 server_base.cc:1034] running on GCE node
W20260806 16:08:06.263370 7362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:06.264091 7357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:06.264786 7357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:06.265983 7357 hybrid_clock.cc:648] HybridClock initialized: now 1786032486265945 us; error 45 us; skew 500 ppm
Aug 06 16:08:06 dist-test-slave-7kzw krb5kdc[7203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032486, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:06.269001 7357 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:08:06.269878 7357 webserver.cc:460] Webserver started at http://127.13.148.1:40605/ using document root <none> and password file <none>
I20260806 16:08:06.270505 7357 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:06.270605 7357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:06.270846 7357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:06.273036 7357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "1b4db043f7b843358d24e81167838ff3"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "65749828138cd9ce1a21d0cd3e4f4865"
server_key_iv: "adde9c15aa481c15ca4276de4ee4d795"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.273605 7357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "1b4db043f7b843358d24e81167838ff3"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "65749828138cd9ce1a21d0cd3e4f4865"
server_key_iv: "adde9c15aa481c15ca4276de4ee4d795"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.277360 7357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 16:08:06.279917 7375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.281126 7357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:08:06.281256 7357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "1b4db043f7b843358d24e81167838ff3"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "65749828138cd9ce1a21d0cd3e4f4865"
server_key_iv: "adde9c15aa481c15ca4276de4ee4d795"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.281363 7357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:06.300635 7357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:06.303735 7357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:06.303973 7357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:06.304602 7357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:06.305441 7357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:06.305526 7357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.305626 7357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:06.305673 7357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.314987 7357 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:37187
I20260806 16:08:06.315053 7535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:37187 every 8 connection(s)
I20260806 16:08:06.315923 7357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:08:06.319792 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 7357
I20260806 16:08:06.319948 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:08:06.320230 13904 external_mini_cluster.cc:1428] Setting key 4f5eb20239a6f3e4300bfae71465624f
Aug 06 16:08:06 dist-test-slave-7kzw krb5kdc[7203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032486, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:06.330790 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.317785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:45521 (local address 127.13.148.62:42583)
0806 16:08:06.317907 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.317912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.318863 (+ 951us) server_negotiation.cc:407] Connection header received
0806 16:08:06.320300 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.320304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.320366 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.320445 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:06.321894 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.322432 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.323605 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.323855 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.323966 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.326750 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:06.326753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:06.326755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:06.326779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:06.328483 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.329076 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.329080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.329082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.329182 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.329581 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.329585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.329586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.329786 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:06.329905 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.330422 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.330590 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260806 16:08:06.331596 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.318159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42583 (local address 127.13.148.1:45521)
0806 16:08:06.318691 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.318734 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:06.320084 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:06.320632 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:06.320642 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:06.321063 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:06.321728 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.321752 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.322587 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.322597 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.323473 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.323483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.324003 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.324007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.324188 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.324816 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:06.324833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:06.326512 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:06.328635 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:06.328642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:06.328657 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:06.328954 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:06.329297 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:06.329300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:06.329302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:06.329456 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:06.329903 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:06.329910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:06.330162 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 16:08:06.330359 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 16:08:06.330738 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":111,"threads_started":1}
I20260806 16:08:06.332839 7538 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42583
I20260806 16:08:06.333132 7538 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:08:06.333655 7538 heartbeater.cc:511] Master 127.13.148.62:42583 requested a full tablet report, sending...
I20260806 16:08:06.335251 7271 ts_manager.cc:194] Registered new tserver with Master: 1b4db043f7b843358d24e81167838ff3 (127.13.148.1:37187)
I20260806 16:08:06.336412 7271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:45521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:08:06.350445 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42583
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:06.465590 7547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:06.465863 7547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:06.466019 7547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:06.470269 7547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:06.470330 7547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:06.470418 7547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:08:06.476056 7547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:06.477125 7547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:06.478134 7547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:06.488368 7554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:06.488457 7547 server_base.cc:1034] running on GCE node
W20260806 16:08:06.488396 7553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:06.488726 7556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:06.489147 7547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:06.489807 7547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:06.491016 7547 hybrid_clock.cc:648] HybridClock initialized: now 1786032486491004 us; error 33 us; skew 500 ppm
Aug 06 16:08:06 dist-test-slave-7kzw krb5kdc[7203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032486, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:06.493978 7547 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:08:06.495059 7547 webserver.cc:460] Webserver started at http://127.13.148.2:38685/ using document root <none> and password file <none>
I20260806 16:08:06.495618 7547 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:06.495679 7547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:06.495851 7547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:06.497915 7547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "63e03cd3dcd44f72895bb85b0149b1e8"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "03dbb2197a0671c7ccb4e9fbdae253fa"
server_key_iv: "2a4f73528d5d4576f4d582d98286cdad"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.498451 7547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "63e03cd3dcd44f72895bb85b0149b1e8"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "03dbb2197a0671c7ccb4e9fbdae253fa"
server_key_iv: "2a4f73528d5d4576f4d582d98286cdad"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.502303 7547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 16:08:06.505111 7564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.506340 7547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:06.506476 7547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "63e03cd3dcd44f72895bb85b0149b1e8"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "03dbb2197a0671c7ccb4e9fbdae253fa"
server_key_iv: "2a4f73528d5d4576f4d582d98286cdad"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.506609 7547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:06.535837 7547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:06.539335 7547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:06.539551 7547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:06.540164 7547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:06.541019 7547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:06.541069 7547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.541134 7547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:06.541172 7547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.550169 7547 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:37407
I20260806 16:08:06.550199 7719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:37407 every 8 connection(s)
I20260806 16:08:06.551141 7547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:08:06.558274 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 7547
I20260806 16:08:06.558436 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:08:06.558789 13904 external_mini_cluster.cc:1428] Setting key 29f19833502c5bede69ec3d1f0c879d0
Aug 06 16:08:06 dist-test-slave-7kzw krb5kdc[7203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032486, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:06.565274 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.553056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:40209 (local address 127.13.148.62:42583)
0806 16:08:06.553204 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.553208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.553976 (+ 768us) server_negotiation.cc:407] Connection header received
0806 16:08:06.555375 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.555378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.555436 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.555517 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:06.556642 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.557178 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.558445 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.558769 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.558885 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.561304 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:06.561308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:06.561310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:06.561336 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:06.563073 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.563570 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.563576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.563580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.563685 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.564035 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.564038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.564039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.564250 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:06.564329 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.564972 (+ 643us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.565115 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:08:06.566603 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.553405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42583 (local address 127.13.148.2:40209)
0806 16:08:06.553827 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.553869 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:06.555163 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:06.555683 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:06.555690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:06.556032 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:06.556495 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.556508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.557304 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.557312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.558318 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.558329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.558923 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.558927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.559098 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.559702 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:06.559718 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:06.561145 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:06.563196 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:06.563201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:06.563212 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:06.563473 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:06.563782 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:06.563785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:06.563786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:06.563922 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:06.564357 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:06.564367 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:06.564705 (+ 338us) client_negotiation.cc:769] Sending connection context
0806 16:08:06.564923 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 16:08:06.565302 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260806 16:08:06.567914 7720 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42583
I20260806 16:08:06.568120 7720 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:06.568557 7720 heartbeater.cc:511] Master 127.13.148.62:42583 requested a full tablet report, sending...
I20260806 16:08:06.569583 7271 ts_manager.cc:194] Registered new tserver with Master: 63e03cd3dcd44f72895bb85b0149b1e8 (127.13.148.2:37407)
I20260806 16:08:06.570442 7271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:40209
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:08:06.593371 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42583
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:06.709324 7727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:06.709661 7727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:06.709709 7727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:06.713785 7727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:06.713850 7727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:06.713907 7727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:08:06.719156 7727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:06.720124 7727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:06.721083 7727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:06.730697 7733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:06.730734 7727 server_base.cc:1034] running on GCE node
W20260806 16:08:06.730726 7734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:06.730726 7736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:06.731321 7727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:06.731956 7727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:06.733143 7727 hybrid_clock.cc:648] HybridClock initialized: now 1786032486733131 us; error 30 us; skew 500 ppm
Aug 06 16:08:06 dist-test-slave-7kzw krb5kdc[7203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032486, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:06.736130 7727 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:08:06.736987 7727 webserver.cc:460] Webserver started at http://127.13.148.3:44597/ using document root <none> and password file <none>
I20260806 16:08:06.737533 7727 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:06.737664 7727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:06.737896 7727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:06.739882 7727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "6bad7040c1be41588b210c3849415df6"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "6a5d92c65a6d85230da08aceba49467a"
server_key_iv: "606327adf163ae8c210784f1f4990d47"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.740423 7727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "6bad7040c1be41588b210c3849415df6"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "6a5d92c65a6d85230da08aceba49467a"
server_key_iv: "606327adf163ae8c210784f1f4990d47"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.744143 7727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:08:06.746588 7745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.747783 7727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:06.747920 7727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "6bad7040c1be41588b210c3849415df6"
format_stamp: "Formatted at 2026-08-06 16:08:06 on dist-test-slave-7kzw"
server_key: "6a5d92c65a6d85230da08aceba49467a"
server_key_iv: "606327adf163ae8c210784f1f4990d47"
server_key_version: "encryptionkey@0"
I20260806 16:08:06.748028 7727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:06.761476 7727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:06.764801 7727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:06.765084 7727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:06.765759 7727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:06.766793 7727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:06.766858 7727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.766927 7727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:06.766958 7727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:06.775795 7727 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:32977
I20260806 16:08:06.775866 7906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:32977 every 8 connection(s)
I20260806 16:08:06.776691 7727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:08:06.781414 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 7727
I20260806 16:08:06.781576 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:08:06.781872 13904 external_mini_cluster.cc:1428] Setting key 4077b8ec7047af09278aa0e490636c50
Aug 06 16:08:06 dist-test-slave-7kzw krb5kdc[7203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032486, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:06.791863 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.778665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:38025 (local address 127.13.148.62:42583)
0806 16:08:06.778823 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.778828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.779739 (+ 911us) server_negotiation.cc:407] Connection header received
0806 16:08:06.781235 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.781239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.781304 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.781416 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:06.783043 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.783731 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.785046 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.785380 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.785489 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.788246 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:06.788250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:06.788253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:06.788284 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:06.789766 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.790299 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.790302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.790304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.790394 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.790767 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.790770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.790772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.790968 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:06.791073 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.791568 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.791691 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:08:06.792658 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.778961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42583 (local address 127.13.148.3:38025)
0806 16:08:06.779532 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.779597 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:06.780988 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:06.781598 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:06.781605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:06.782097 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:06.782896 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.782918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.783867 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.783870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.784900 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.784911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.785529 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.785534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.785736 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.786294 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:06.786312 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:06.788070 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:06.789896 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:06.789902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:06.789917 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:06.790190 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:06.790510 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:06.790513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:06.790516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:06.790665 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:06.791072 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:06.791080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:06.791328 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 16:08:06.791506 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 16:08:06.791840 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
I20260806 16:08:06.793864 7907 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42583
I20260806 16:08:06.794085 7907 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:06.794585 7907 heartbeater.cc:511] Master 127.13.148.62:42583 requested a full tablet report, sending...
I20260806 16:08:06.795648 7271 ts_manager.cc:194] Registered new tserver with Master: 6bad7040c1be41588b210c3849415df6 (127.13.148.3:32977)
I20260806 16:08:06.796257 7271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:38025
I20260806 16:08:06.807945 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:08:06.818135 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.810166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.13.148.62:42583)
0806 16:08:06.810305 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.810309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.810595 (+ 286us) server_negotiation.cc:407] Connection header received
0806 16:08:06.810673 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.810676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.810725 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.810851 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:06.811574 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.812104 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.812906 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.813129 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.813217 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.814920 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:06.814923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:06.814925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:06.814953 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:06.816368 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.816935 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.816939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.816940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.817045 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:06.817344 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:06.817346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:06.817348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:06.817554 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:06.817670 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.817925 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.818008 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:08:06.821146 7271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:08:06.824484 7271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:08:06.839529 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.833368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.13.148.1:37187)
0806 16:08:06.834151 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.834158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.834189 (+ 31us) server_negotiation.cc:407] Connection header received
0806 16:08:06.834276 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.834284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.834517 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.834704 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:06.834996 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.836221 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.837896 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.838784 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.838896 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.839045 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.839268 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.839366 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":111,"threads_started":1}
I20260806 16:08:06.839543 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.833150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37187 (local address 127.0.0.1:46304)
0806 16:08:06.833937 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.833995 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:06.834134 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:06.834698 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:06.834705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:06.834729 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:06.834882 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.834891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.836352 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.836355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.837748 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.837760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.838917 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.838921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.839051 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.839095 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 16:08:06.839224 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 16:08:06.839366 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260806 16:08:06.839740 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.833179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.13.148.2:37407)
0806 16:08:06.833951 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.833966 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.834284 (+ 318us) server_negotiation.cc:407] Connection header received
0806 16:08:06.834368 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.834371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.834576 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.834730 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:06.835463 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.836588 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.837892 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.838795 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.838932 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.839087 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.839444 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.839560 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":132,"threads_started":1}
I20260806 16:08:06.840219 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.833060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37407 (local address 127.0.0.1:56574)
0806 16:08:06.833936 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.833994 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:06.834135 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:06.834901 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:06.834904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:06.834922 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:06.835262 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.835271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.836744 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.836754 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.837731 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.837743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.839156 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.839163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.839275 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.839298 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:08:06.839970 (+ 672us) client_negotiation.cc:240] Negotiation successful
0806 16:08:06.840056 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":241,"thread_start_us":95,"threads_started":1}
I20260806 16:08:06.841224 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.834391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.13.148.3:32977)
0806 16:08:06.835170 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.835182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.836509 (+ 1327us) server_negotiation.cc:407] Connection header received
0806 16:08:06.836655 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.836661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.836894 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.837057 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:06.837307 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.838422 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.839875 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.840619 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.840736 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.840885 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.841001 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.841086 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":210,"threads_started":1}
I20260806 16:08:06.841849 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.833909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:32977 (local address 127.0.0.1:36690)
0806 16:08:06.836388 (+ 2479us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.836405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:06.836509 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:06.837065 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:06.837068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:06.837091 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:06.837177 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.837183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.838604 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.838608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.839704 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.839720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.840752 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.840756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.840849 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.840876 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 16:08:06.840975 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:08:06.841707 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":106,"threads_started":1}
I20260806 16:08:06.843185 7443 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcb1f529f2443ee8f05320a98d55c81 (DEFAULT_TABLE table=test-table [id=18f696f068f84e7d8cac82116bef55c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:06.843283 7632 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcb1f529f2443ee8f05320a98d55c81 (DEFAULT_TABLE table=test-table [id=18f696f068f84e7d8cac82116bef55c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:06.844246 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcb1f529f2443ee8f05320a98d55c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:06.844516 7632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcb1f529f2443ee8f05320a98d55c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:06.844774 7805 tablet_service.cc:1467] Processing CreateTablet for tablet 1bcb1f529f2443ee8f05320a98d55c81 (DEFAULT_TABLE table=test-table [id=18f696f068f84e7d8cac82116bef55c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:06.845824 7805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bcb1f529f2443ee8f05320a98d55c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:06.851374 7934 tablet_bootstrap.cc:492] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3: Bootstrap starting.
I20260806 16:08:06.852494 7935 tablet_bootstrap.cc:492] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: Bootstrap starting.
I20260806 16:08:06.854074 7936 tablet_bootstrap.cc:492] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: Bootstrap starting.
I20260806 16:08:06.854120 7934 tablet_bootstrap.cc:654] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:06.855183 7935 tablet_bootstrap.cc:654] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:06.855146 7934 log.cc:824] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:06.856014 7935 log.cc:824] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:06.856161 7936 tablet_bootstrap.cc:654] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:06.857559 7936 log.cc:824] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:06.858186 7934 tablet_bootstrap.cc:492] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3: No bootstrap required, opened a new log
I20260806 16:08:06.858201 7935 tablet_bootstrap.cc:492] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: No bootstrap required, opened a new log
I20260806 16:08:06.858417 7934 ts_tablet_manager.cc:1397] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:08:06.858445 7935 ts_tablet_manager.cc:1397] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:08:06.859812 7936 tablet_bootstrap.cc:492] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: No bootstrap required, opened a new log
I20260806 16:08:06.860152 7936 ts_tablet_manager.cc:1397] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 16:08:06.862357 7934 raft_consensus.cc:348] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.862711 7934 raft_consensus.cc:374] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:06.862795 7934 raft_consensus.cc:729] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4db043f7b843358d24e81167838ff3, State: Initialized, Role: FOLLOWER
I20260806 16:08:06.862665 7935 raft_consensus.cc:348] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.862947 7935 raft_consensus.cc:374] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:06.863009 7935 raft_consensus.cc:729] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e03cd3dcd44f72895bb85b0149b1e8, State: Initialized, Role: FOLLOWER
I20260806 16:08:06.863341 7934 consensus_queue.cc:260] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.863477 7935 consensus_queue.cc:260] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.864673 7934 ts_tablet_manager.cc:1428] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:08:06.864468 7936 raft_consensus.cc:348] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.864717 7935 ts_tablet_manager.cc:1428] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:08:06.864775 7936 raft_consensus.cc:374] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:06.864849 7936 raft_consensus.cc:729] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bad7040c1be41588b210c3849415df6, State: Initialized, Role: FOLLOWER
I20260806 16:08:06.865079 7538 heartbeater.cc:503] Master 127.13.148.62:42583 was elected leader, sending a full tablet report...
I20260806 16:08:06.865321 7936 consensus_queue.cc:260] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.865509 7720 heartbeater.cc:503] Master 127.13.148.62:42583 was elected leader, sending a full tablet report...
I20260806 16:08:06.866320 7936 ts_tablet_manager.cc:1428] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 16:08:06.866515 7907 heartbeater.cc:503] Master 127.13.148.62:42583 was elected leader, sending a full tablet report...
I20260806 16:08:06.872480 7944 raft_consensus.cc:482] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:08:06.872713 7944 raft_consensus.cc:504] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.873793 7944 leader_election.cc:288] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b4db043f7b843358d24e81167838ff3 (127.13.148.1:37187), 6bad7040c1be41588b210c3849415df6 (127.13.148.3:32977)
I20260806 16:08:06.878005 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.874570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:49925 (local address 127.13.148.3:32977)
0806 16:08:06.874763 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.874767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.874818 (+ 51us) server_negotiation.cc:407] Connection header received
0806 16:08:06.874915 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.874918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.874970 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.875047 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:06.875360 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.875877 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.877008 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.877403 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.877536 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.877591 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.877795 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.877866 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 16:08:06.878016 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.874147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:40013 (local address 127.13.148.1:37187)
0806 16:08:06.874289 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.874295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.874313 (+ 18us) server_negotiation.cc:407] Connection header received
0806 16:08:06.874439 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.874445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.874536 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.874627 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:06.875076 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.875724 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.876884 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.877400 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.877532 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.877588 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.877829 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.877910 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:08:06.878006 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.874059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37187 (local address 127.13.148.2:40013)
0806 16:08:06.874207 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.874227 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:06.874337 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:06.874766 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:06.874769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:06.874791 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:06.874911 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.874917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.875835 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.875839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.876739 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.876750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.877536 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.877539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.877686 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.877703 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:08:06.877793 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 16:08:06.877869 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":30}
I20260806 16:08:06.878024 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.874366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:32977 (local address 127.13.148.2:49925)
0806 16:08:06.874753 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.874765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:06.874847 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:06.875083 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:06.875086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:06.875105 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:06.875231 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.875237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.876015 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.876017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.876875 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:06.876885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.877526 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.877530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:06.877671 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.877692 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 16:08:06.877755 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 16:08:06.877851 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":18,"thread_start_us":178,"threads_started":1}
I20260806 16:08:06.879030 7841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcb1f529f2443ee8f05320a98d55c81" candidate_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bad7040c1be41588b210c3849415df6" is_pre_election: true
I20260806 16:08:06.879030 7468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcb1f529f2443ee8f05320a98d55c81" candidate_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4db043f7b843358d24e81167838ff3" is_pre_election: true
I20260806 16:08:06.879375 7841 raft_consensus.cc:2454] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63e03cd3dcd44f72895bb85b0149b1e8 in term 0.
I20260806 16:08:06.879375 7468 raft_consensus.cc:2454] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63e03cd3dcd44f72895bb85b0149b1e8 in term 0.
I20260806 16:08:06.879873 7571 leader_election.cc:302] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4db043f7b843358d24e81167838ff3, 63e03cd3dcd44f72895bb85b0149b1e8; no voters:
I20260806 16:08:06.880141 7944 raft_consensus.cc:2781] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:08:06.880244 7944 raft_consensus.cc:482] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:08:06.880309 7944 raft_consensus.cc:3037] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:06.881382 7944 raft_consensus.cc:504] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.881753 7944 leader_election.cc:288] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [CANDIDATE]: Term 1 election: Requested vote from peers 1b4db043f7b843358d24e81167838ff3 (127.13.148.1:37187), 6bad7040c1be41588b210c3849415df6 (127.13.148.3:32977)
I20260806 16:08:06.882157 7468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcb1f529f2443ee8f05320a98d55c81" candidate_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4db043f7b843358d24e81167838ff3"
I20260806 16:08:06.882238 7841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bcb1f529f2443ee8f05320a98d55c81" candidate_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bad7040c1be41588b210c3849415df6"
I20260806 16:08:06.882302 7468 raft_consensus.cc:3037] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:06.882401 7841 raft_consensus.cc:3037] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:06.883662 7468 raft_consensus.cc:2454] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63e03cd3dcd44f72895bb85b0149b1e8 in term 1.
I20260806 16:08:06.883868 7841 raft_consensus.cc:2454] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63e03cd3dcd44f72895bb85b0149b1e8 in term 1.
I20260806 16:08:06.884001 7571 leader_election.cc:302] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4db043f7b843358d24e81167838ff3, 63e03cd3dcd44f72895bb85b0149b1e8; no voters:
I20260806 16:08:06.884183 7944 raft_consensus.cc:2781] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:06.884450 7944 raft_consensus.cc:686] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 1 LEADER]: Becoming Leader. State: Replica: 63e03cd3dcd44f72895bb85b0149b1e8, State: Running, Role: LEADER
I20260806 16:08:06.884819 7944 consensus_queue.cc:237] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } }
I20260806 16:08:06.888415 7269 catalog_manager.cc:5495] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63e03cd3dcd44f72895bb85b0149b1e8 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e03cd3dcd44f72895bb85b0149b1e8" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 } health_report { overall_health: UNKNOWN } } }
I20260806 16:08:06.905544 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.901581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.13.148.2:37407)
0806 16:08:06.901733 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:06.901737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:06.901753 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:08:06.901924 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:06.901926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:06.901986 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:06.902102 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:06.902362 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.902937 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.903940 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:06.904180 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:06.904290 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:06.904642 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:06.905057 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:06.905228 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:06.905336 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 16:08:06.905378 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 16:08:06.911988 7841 raft_consensus.cc:1261] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 1 FOLLOWER]: Refusing update from remote peer 63e03cd3dcd44f72895bb85b0149b1e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:06.912089 7468 raft_consensus.cc:1261] T 1bcb1f529f2443ee8f05320a98d55c81 P 1b4db043f7b843358d24e81167838ff3 [term 1 FOLLOWER]: Refusing update from remote peer 63e03cd3dcd44f72895bb85b0149b1e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:06.912854 7944 consensus_queue.cc:1035] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bad7040c1be41588b210c3849415df6" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 32977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:06.913074 7950 consensus_queue.cc:1035] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4db043f7b843358d24e81167838ff3" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:06.921705 7955 mvcc.cc:205] Tried to move back new op lower bound from 7315589066383577088 to 7315589066283126784. Current Snapshot: MvccSnapshot[applied={T|T < 7315589066383577088}]
I20260806 16:08:06.923883 7954 mvcc.cc:205] Tried to move back new op lower bound from 7315589066383577088 to 7315589066283126784. Current Snapshot: MvccSnapshot[applied={T|T < 7315589066383577088 or (T in {7315589066383577088})}]
I20260806 16:08:06.929953 7269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:06.930174 7269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:06.933087 7269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P daedf4d2d3d24b2cb22ea10800a5c7b7: Sending DeleteTablet for 3 replicas of tablet 1bcb1f529f2443ee8f05320a98d55c81
I20260806 16:08:06.934067 7632 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcb1f529f2443ee8f05320a98d55c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:06 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:56574
I20260806 16:08:06.934077 7443 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcb1f529f2443ee8f05320a98d55c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:06 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46304
I20260806 16:08:06.934757 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 7357
I20260806 16:08:06.934880 7805 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bcb1f529f2443ee8f05320a98d55c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:06 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36690
I20260806 16:08:06.957742 7962 tablet_replica.cc:330] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: stopping tablet replica
I20260806 16:08:06.958417 7962 raft_consensus.cc:2229] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:08:06.959244 7962 raft_consensus.cc:2258] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:06.961619 7962 ts_tablet_manager.cc:1905] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:06.964879 7966 tablet_replica.cc:330] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: stopping tablet replica
I20260806 16:08:06.965386 7966 raft_consensus.cc:2229] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:06.965396 7962 ts_tablet_manager.cc:1918] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:06.965621 7962 log.cc:1196] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/1bcb1f529f2443ee8f05320a98d55c81
I20260806 16:08:06.965891 7966 raft_consensus.cc:2258] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:06.966265 7962 ts_tablet_manager.cc:1939] T 1bcb1f529f2443ee8f05320a98d55c81 P 63e03cd3dcd44f72895bb85b0149b1e8: Deleting consensus metadata
I20260806 16:08:06.967906 7249 catalog_manager.cc:4841] TS 63e03cd3dcd44f72895bb85b0149b1e8 (127.13.148.2:37407): tablet 1bcb1f529f2443ee8f05320a98d55c81 (table test-table [id=18f696f068f84e7d8cac82116bef55c7]) successfully deleted
I20260806 16:08:06.968250 7966 ts_tablet_manager.cc:1905] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:06.970698 7966 ts_tablet_manager.cc:1918] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:06.970825 7966 log.cc:1196] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/1bcb1f529f2443ee8f05320a98d55c81
I20260806 16:08:06.971158 7966 ts_tablet_manager.cc:1939] T 1bcb1f529f2443ee8f05320a98d55c81 P 6bad7040c1be41588b210c3849415df6: Deleting consensus metadata
I20260806 16:08:06.972400 7254 catalog_manager.cc:4841] TS 6bad7040c1be41588b210c3849415df6 (127.13.148.3:32977): tablet 1bcb1f529f2443ee8f05320a98d55c81 (table test-table [id=18f696f068f84e7d8cac82116bef55c7]) successfully deleted
W20260806 16:08:06.975500 7254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:37187 (error 108)
I20260806 16:08:06.975625 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 7547
I20260806 16:08:06.979903 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:06.978699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:37187 (local address 127.0.0.1:46312)
0806 16:08:06.979711 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:06.979794 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":892}
W20260806 16:08:06.981782 7254 catalog_manager.cc:4584] TS 1b4db043f7b843358d24e81167838ff3 (127.13.148.1:37187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bcb1f529f2443ee8f05320a98d55c81: Network error: Client connection negotiation failed: client connection to 127.13.148.1:37187: connect: Connection refused (error 111)
I20260806 16:08:06.987568 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 7727
I20260806 16:08:07.001900 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 7224
2026-08-06T16:08:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2536 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T16:08:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:07Z Disabled control of system clock
I20260806 16:08:07.040512 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44871
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:44871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:08:07.153381 7973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:07.153671 7973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:07.153710 7973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:07.158453 7973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:07.158573 7973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:07.158591 7973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:07.158624 7973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:07.158640 7973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:07.163520 7973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:44871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:07.164614 7973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:07.165987 7973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:07.174649 7980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:07.174726 7983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:07.174697 7979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:07.174703 7973 server_base.cc:1034] running on GCE node
I20260806 16:08:07.175555 7973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:07.176730 7973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:07.177897 7973 hybrid_clock.cc:648] HybridClock initialized: now 1786032487177887 us; error 31 us; skew 500 ppm
I20260806 16:08:07.179829 7973 webserver.cc:460] Webserver started at http://127.13.148.62:40251/ using document root <none> and password file <none>
I20260806 16:08:07.180379 7973 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:07.180441 7973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:07.180608 7973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:07.182678 7973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "077b53f652614633a40b63217965e61e"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "74e6970bb6234ac48e1daa662a6d9db2"
server_key_iv: "cd0039886143d42a837ef984955f6d6a"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.183223 7973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "077b53f652614633a40b63217965e61e"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "74e6970bb6234ac48e1daa662a6d9db2"
server_key_iv: "cd0039886143d42a837ef984955f6d6a"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.186806 7973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:08:07.189153 7990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.190351 7973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:07.190511 7973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "077b53f652614633a40b63217965e61e"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "74e6970bb6234ac48e1daa662a6d9db2"
server_key_iv: "cd0039886143d42a837ef984955f6d6a"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.190632 7973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:07.203495 7973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:07.204272 7973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:07.204514 7973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:07.212050 7973 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:44871
I20260806 16:08:07.212100 8057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:44871 every 8 connection(s)
I20260806 16:08:07.213030 7973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:07.216531 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:07.216941 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 7973
I20260806 16:08:07.217092 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:07.217422 13904 external_mini_cluster.cc:1428] Setting key 5eccbd219c0960eea437804c0047b798
I20260806 16:08:07.223264 8058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e: Bootstrap starting.
I20260806 16:08:07.225782 8058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:07.226675 8058 log.cc:824] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:07.227345 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.219067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.13.148.62:44871)
0806 16:08:07.219786 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:07.219802 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:07.219837 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:08:07.220483 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:07.220500 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:07.220931 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:07.221321 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:07.221762 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.222984 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.224025 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.224484 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.224668 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.225559 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:07.225564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:07.225577 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:07.226010 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:07.226066 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:07.226087 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:07.226191 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:07.226338 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 16:08:07.226619 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":153,"threads_started":1}
I20260806 16:08:07.228814 8058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e: No bootstrap required, opened a new log
I20260806 16:08:07.232627 8058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077b53f652614633a40b63217965e61e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44871 } }
I20260806 16:08:07.232848 8058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:07.232894 8058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077b53f652614633a40b63217965e61e, State: Initialized, Role: FOLLOWER
I20260806 16:08:07.233410 8058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077b53f652614633a40b63217965e61e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44871 } }
I20260806 16:08:07.233583 8058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:07.233667 8058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:07.233767 8058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:07.234764 8058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077b53f652614633a40b63217965e61e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44871 } }
I20260806 16:08:07.235077 8058 leader_election.cc:302] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077b53f652614633a40b63217965e61e; no voters:
I20260806 16:08:07.235343 8058 leader_election.cc:288] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:07.235476 8063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:07.235743 8063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [term 1 LEADER]: Becoming Leader. State: Replica: 077b53f652614633a40b63217965e61e, State: Running, Role: LEADER
I20260806 16:08:07.236052 8063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077b53f652614633a40b63217965e61e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44871 } }
I20260806 16:08:07.236562 8058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:07.237746 8066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077b53f652614633a40b63217965e61e. Latest consensus state: current_term: 1 leader_uuid: "077b53f652614633a40b63217965e61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077b53f652614633a40b63217965e61e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44871 } } }
I20260806 16:08:07.237906 8066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:07.238214 8064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "077b53f652614633a40b63217965e61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077b53f652614633a40b63217965e61e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44871 } } }
I20260806 16:08:07.238317 8064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:07.238422 8075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:07.241607 8075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:07.246955 8075 catalog_manager.cc:1347] Generated new cluster ID: 39c783642cad476daec6d4d13129e5de
I20260806 16:08:07.247025 8075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:07.258045 8075 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:07.259008 8075 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:07.268739 8075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e: Generated new TSK 0
I20260806 16:08:07.269547 8075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:07.276127 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44871
--builtin_ntp_servers=127.13.148.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:08:07.391726 8088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:07.392040 8088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:07.392120 8088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:07.396376 8088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:07.396453 8088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:07.396543 8088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:08:07.401464 8088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:07.402664 8088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:07.404084 8088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:07.414124 8097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:07.414117 8096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:07.414204 8100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:07.414309 8088 server_base.cc:1034] running on GCE node
I20260806 16:08:07.414923 8088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:07.415645 8088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:07.416869 8088 hybrid_clock.cc:648] HybridClock initialized: now 1786032487416862 us; error 38 us; skew 500 ppm
I20260806 16:08:07.419036 8088 webserver.cc:460] Webserver started at http://127.13.148.1:35625/ using document root <none> and password file <none>
I20260806 16:08:07.419694 8088 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:07.419795 8088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:07.420047 8088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:07.422259 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "9bb9f10aa634458b94725db646a8399e"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "a994c353168891536a3a4f8fa9f7960a"
server_key_iv: "9662e07c75b83d3db54f91bacce94cfa"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.422920 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "9bb9f10aa634458b94725db646a8399e"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "a994c353168891536a3a4f8fa9f7960a"
server_key_iv: "9662e07c75b83d3db54f91bacce94cfa"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.426935 8088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:07.429613 8106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.430912 8088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:07.431088 8088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "9bb9f10aa634458b94725db646a8399e"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "a994c353168891536a3a4f8fa9f7960a"
server_key_iv: "9662e07c75b83d3db54f91bacce94cfa"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.431226 8088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:07.451599 8088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:07.452394 8088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:07.452613 8088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:07.453274 8088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:07.454314 8088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:07.454391 8088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.454497 8088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:07.454547 8088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.463891 8088 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:46707
I20260806 16:08:07.463945 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:46707 every 8 connection(s)
I20260806 16:08:07.464815 8088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:08:07.474272 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.466754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:51503 (local address 127.13.148.62:44871)
0806 16:08:07.466903 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:07.466906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:07.467968 (+ 1062us) server_negotiation.cc:407] Connection header received
0806 16:08:07.469393 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:07.469396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:07.469445 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:07.469522 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:07.470424 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.471031 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.472231 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.472446 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.472528 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.473726 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:07.473729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:07.473731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:07.473745 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:07.473765 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:07.473773 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:07.473840 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:07.474011 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 16:08:07.474086 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:08:07.474272 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 8088
I20260806 16:08:07.474397 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:08:07.474713 13904 external_mini_cluster.cc:1428] Setting key 83bee9793ca2bb79401065a583ddbc20
I20260806 16:08:07.475188 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.467148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44871 (local address 127.13.148.1:51503)
0806 16:08:07.467823 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:07.467868 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:07.469181 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:07.469702 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:07.469709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:07.469792 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:07.470293 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.470306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.471164 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.471167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.472056 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.472065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.472617 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.472621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.472772 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.473429 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:07.473430 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:07.473472 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:07.473537 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:07.473855 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:07.473862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:07.473867 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:08:07.473971 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:08:07.474194 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260806 16:08:07.476511 8263 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44871
I20260806 16:08:07.476769 8263 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:07.476922 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44871
--builtin_ntp_servers=127.13.148.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:08:07.477360 8263 heartbeater.cc:511] Master 127.13.148.62:44871 requested a full tablet report, sending...
I20260806 16:08:07.479060 8011 ts_manager.cc:194] Registered new tserver with Master: 9bb9f10aa634458b94725db646a8399e (127.13.148.1:46707)
I20260806 16:08:07.480274 8011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.1:51503
W20260806 16:08:07.592499 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:07.592801 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:07.592841 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:07.597059 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:07.597121 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:07.597196 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:08:07.601981 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:07.603235 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:07.604760 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:07.615010 8276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:07.614985 8272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:07.614985 8273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:07.615345 8266 server_base.cc:1034] running on GCE node
I20260806 16:08:07.615758 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:07.616431 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:07.617674 8266 hybrid_clock.cc:648] HybridClock initialized: now 1786032487617644 us; error 43 us; skew 500 ppm
I20260806 16:08:07.619668 8266 webserver.cc:460] Webserver started at http://127.13.148.2:44519/ using document root <none> and password file <none>
I20260806 16:08:07.620275 8266 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:07.620369 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:07.620602 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:07.622788 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "30b1368a51474598814ddaff16f63947"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "69f870db597a90578df076d923b5cc53"
server_key_iv: "478606858549a3701b8e6b5e2ce84f44"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.623363 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "30b1368a51474598814ddaff16f63947"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "69f870db597a90578df076d923b5cc53"
server_key_iv: "478606858549a3701b8e6b5e2ce84f44"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.627230 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:08:07.629667 8287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.630928 8266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:07.631059 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "30b1368a51474598814ddaff16f63947"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "69f870db597a90578df076d923b5cc53"
server_key_iv: "478606858549a3701b8e6b5e2ce84f44"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.631170 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:07.643524 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:07.644198 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:07.644405 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:07.645009 8266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:07.646003 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:07.646073 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.646148 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:07.646198 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.654794 8266 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:41921
I20260806 16:08:07.654881 8438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:41921 every 8 connection(s)
I20260806 16:08:07.655731 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:08:07.664913 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 8266
I20260806 16:08:07.665043 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:08:07.665395 13904 external_mini_cluster.cc:1428] Setting key 43d25af17350ba7da7da5cf3099fe679
I20260806 16:08:07.667500 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.657704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:34609 (local address 127.13.148.62:44871)
0806 16:08:07.657858 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:07.657863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:07.659131 (+ 1268us) server_negotiation.cc:407] Connection header received
0806 16:08:07.660569 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:07.660572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:07.660625 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:07.660717 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:07.661910 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.662890 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.664805 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.665166 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.665298 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.666790 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:07.666793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:07.666796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:07.666810 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:07.666855 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:07.666866 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:07.666941 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:07.667172 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 16:08:07.667283 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:08:07.668496 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.658144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44871 (local address 127.13.148.2:34609)
0806 16:08:07.658968 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:07.659011 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:07.660342 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:07.660887 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:07.660894 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:07.660977 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:07.661699 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.661723 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.663060 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.663067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.664665 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.664679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.665391 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.665398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.665703 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.666441 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:07.666444 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:07.666526 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:07.666614 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:07.666988 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:07.666997 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:07.667004 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 16:08:07.667133 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 16:08:07.667467 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":121,"threads_started":1}
I20260806 16:08:07.668934 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44871
--builtin_ntp_servers=127.13.148.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:08:07.669696 8439 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44871
I20260806 16:08:07.669968 8439 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:07.670560 8439 heartbeater.cc:511] Master 127.13.148.62:44871 requested a full tablet report, sending...
I20260806 16:08:07.671802 8011 ts_manager.cc:194] Registered new tserver with Master: 30b1368a51474598814ddaff16f63947 (127.13.148.2:41921)
I20260806 16:08:07.672410 8011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.2:34609
W20260806 16:08:07.784816 8443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:07.785111 8443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:07.785180 8443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:07.789351 8443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:07.789440 8443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:07.789529 8443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:08:07.794394 8443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:44871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:07.795549 8443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:07.797000 8443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:07.806815 8453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:07.806818 8455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:07.806815 8451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:07.807058 8443 server_base.cc:1034] running on GCE node
I20260806 16:08:07.807539 8443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:07.808257 8443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:07.809449 8443 hybrid_clock.cc:648] HybridClock initialized: now 1786032487809428 us; error 40 us; skew 500 ppm
I20260806 16:08:07.811455 8443 webserver.cc:460] Webserver started at http://127.13.148.3:44061/ using document root <none> and password file <none>
I20260806 16:08:07.812077 8443 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:07.812175 8443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:07.812426 8443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:07.814564 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "d8f1f1ab40cb4be296583921b9d7f2cd"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "18d21168ed62e68fc85a46f02e2ab739"
server_key_iv: "3ffbf02a4eec375756ad6e1f4163be73"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.815147 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f1f1ab40cb4be296583921b9d7f2cd"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "18d21168ed62e68fc85a46f02e2ab739"
server_key_iv: "3ffbf02a4eec375756ad6e1f4163be73"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.818871 8443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:08:07.821381 8462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.822649 8443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:07.822805 8443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "d8f1f1ab40cb4be296583921b9d7f2cd"
format_stamp: "Formatted at 2026-08-06 16:08:07 on dist-test-slave-7kzw"
server_key: "18d21168ed62e68fc85a46f02e2ab739"
server_key_iv: "3ffbf02a4eec375756ad6e1f4163be73"
server_key_version: "encryptionkey@0"
I20260806 16:08:07.822919 8443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:07.845170 8443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:07.845906 8443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:07.846115 8443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:07.846791 8443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:07.847815 8443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:07.847882 8443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.847954 8443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:07.848001 8443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:07.857180 8443 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:40341
I20260806 16:08:07.857206 8616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:40341 every 8 connection(s)
I20260806 16:08:07.858186 8443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:08:07.867255 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.860047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:37863 (local address 127.13.148.62:44871)
0806 16:08:07.860226 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:07.860231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:07.861253 (+ 1022us) server_negotiation.cc:407] Connection header received
0806 16:08:07.862607 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:07.862609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:07.862658 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:07.862737 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:07.863621 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.864135 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.865287 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.865520 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.865604 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.866762 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:07.866765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:07.866767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:07.866781 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:07.866802 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:07.866810 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:07.866880 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:07.867065 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 16:08:07.867131 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 16:08:07.868060 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.860425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44871 (local address 127.13.148.3:37863)
0806 16:08:07.861080 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:07.861120 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:07.862396 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:07.862922 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:07.862929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:07.863008 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:07.863493 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.863506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.864262 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.864265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.865160 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.865170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.865662 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.865666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.865805 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.866399 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:07.866401 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:07.866440 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:07.866576 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:07.866892 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:07.866899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:07.866903 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 16:08:07.867010 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 16:08:07.867234 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260806 16:08:07.868212 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 8443
I20260806 16:08:07.868330 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:08:07.868625 13904 external_mini_cluster.cc:1428] Setting key 32f83b42c748cca5e2706cda04009d13
I20260806 16:08:07.869388 8617 heartbeater.cc:347] Connected to a master server at 127.13.148.62:44871
I20260806 16:08:07.869626 8617 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:07.870208 8617 heartbeater.cc:511] Master 127.13.148.62:44871 requested a full tablet report, sending...
I20260806 16:08:07.871363 8011 ts_manager.cc:194] Registered new tserver with Master: d8f1f1ab40cb4be296583921b9d7f2cd (127.13.148.3:40341)
I20260806 16:08:07.871927 8011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.3:37863
I20260806 16:08:07.883080 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:08:07.889422 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.885192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.13.148.62:44871)
0806 16:08:07.885340 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:07.885345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:07.885517 (+ 172us) server_negotiation.cc:407] Connection header received
0806 16:08:07.885657 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:07.885659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:07.885703 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:07.885770 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:07.886020 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.886577 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.887560 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.887810 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.887904 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.888900 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:07.888903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:07.888905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:07.888917 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:07.888937 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:07.888945 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:07.889050 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:07.889182 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 16:08:07.889250 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:08:07.892766 8011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:08:07.896087 8011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:08:07.912011 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.905552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.13.148.1:46707)
0806 16:08:07.906021 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:07.906034 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:07.906065 (+ 31us) server_negotiation.cc:407] Connection header received
0806 16:08:07.906287 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:07.906297 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:07.906541 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:07.906714 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:07.907260 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.908531 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.910297 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.911192 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.911320 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.911527 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:07.911691 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 16:08:07.911810 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":139,"threads_started":1}
I20260806 16:08:07.912004 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.905428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46707 (local address 127.0.0.1:60614)
0806 16:08:07.905872 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:07.905924 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:07.906047 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:07.906911 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:07.906917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:07.906944 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:07.907101 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.907109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.908634 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.908641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.910120 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.910134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.911344 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.911348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.911469 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.911515 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 16:08:07.911659 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 16:08:07.911811 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260806 16:08:07.912402 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.906327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.13.148.3:40341)
0806 16:08:07.906980 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:07.906994 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:07.907020 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:08:07.907099 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:07.907104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:07.907329 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:07.907489 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:07.907840 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.908792 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.910401 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.911192 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.911321 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.911534 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:07.912088 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 16:08:07.912206 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":139,"threads_started":1}
I20260806 16:08:07.912693 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.905493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:40341 (local address 127.0.0.1:53140)
0806 16:08:07.906369 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:07.906387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:07.906648 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:07.907573 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:07.907577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:07.907595 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:07.907713 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.907719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.908937 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.908940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.910261 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.910272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.911623 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.911630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.911731 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.911760 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 16:08:07.911845 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 16:08:07.911926 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":133,"threads_started":1}
I20260806 16:08:07.912742 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.906659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.13.148.2:41921)
0806 16:08:07.907127 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:07.907139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:07.907393 (+ 254us) server_negotiation.cc:407] Connection header received
0806 16:08:07.907605 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:07.907611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:07.907841 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:07.907959 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:07.908456 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.909440 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.910908 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.911631 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.911737 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.911912 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:07.912446 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 16:08:07.912557 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":148,"threads_started":1}
I20260806 16:08:07.913105 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:07.906199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41921 (local address 127.0.0.1:57280)
0806 16:08:07.907297 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:07.907310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:07.907461 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:07.908115 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:07.908117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:07.908133 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:07.908307 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.908315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.909625 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.909630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.910781 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:07.910792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:07.911981 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:07.911985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:07.912274 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:07.912300 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 16:08:07.912385 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 16:08:07.912545 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":92,"threads_started":1}
I20260806 16:08:07.915676 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 8df170da8110421abc2f02b386759e08 (DEFAULT_TABLE table=test-table [id=6b8d376bd67e4a31b75f1e2d773beafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:07.915843 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 8df170da8110421abc2f02b386759e08 (DEFAULT_TABLE table=test-table [id=6b8d376bd67e4a31b75f1e2d773beafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:07.915843 8532 tablet_service.cc:1467] Processing CreateTablet for tablet 8df170da8110421abc2f02b386759e08 (DEFAULT_TABLE table=test-table [id=6b8d376bd67e4a31b75f1e2d773beafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:07.916806 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df170da8110421abc2f02b386759e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:07.916909 8532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df170da8110421abc2f02b386759e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:07.916910 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df170da8110421abc2f02b386759e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:07.924767 8644 tablet_bootstrap.cc:492] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e: Bootstrap starting.
I20260806 16:08:07.925046 8645 tablet_bootstrap.cc:492] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Bootstrap starting.
I20260806 16:08:07.925242 8647 tablet_bootstrap.cc:492] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Bootstrap starting.
I20260806 16:08:07.927358 8647 tablet_bootstrap.cc:654] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:07.927417 8645 tablet_bootstrap.cc:654] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:07.927417 8644 tablet_bootstrap.cc:654] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:07.928180 8647 log.cc:824] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:07.928366 8644 log.cc:824] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:07.928366 8645 log.cc:824] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:07.930526 8644 tablet_bootstrap.cc:492] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e: No bootstrap required, opened a new log
I20260806 16:08:07.930572 8647 tablet_bootstrap.cc:492] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: No bootstrap required, opened a new log
I20260806 16:08:07.930572 8645 tablet_bootstrap.cc:492] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: No bootstrap required, opened a new log
I20260806 16:08:07.930735 8644 ts_tablet_manager.cc:1397] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 16:08:07.930763 8647 ts_tablet_manager.cc:1397] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:08:07.930761 8645 ts_tablet_manager.cc:1397] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 16:08:07.934053 8644 raft_consensus.cc:348] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:07.934324 8644 raft_consensus.cc:374] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:07.934398 8644 raft_consensus.cc:729] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb9f10aa634458b94725db646a8399e, State: Initialized, Role: FOLLOWER
I20260806 16:08:07.935030 8644 consensus_queue.cc:260] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:07.935524 8645 raft_consensus.cc:348] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:07.935813 8645 raft_consensus.cc:374] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:07.935524 8647 raft_consensus.cc:348] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:07.935885 8645 raft_consensus.cc:729] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b1368a51474598814ddaff16f63947, State: Initialized, Role: FOLLOWER
I20260806 16:08:07.935900 8647 raft_consensus.cc:374] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:07.935969 8647 raft_consensus.cc:729] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f1f1ab40cb4be296583921b9d7f2cd, State: Initialized, Role: FOLLOWER
I20260806 16:08:07.936200 8644 ts_tablet_manager.cc:1428] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 16:08:07.936399 8645 consensus_queue.cc:260] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:07.936411 8647 consensus_queue.cc:260] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:07.936607 8263 heartbeater.cc:503] Master 127.13.148.62:44871 was elected leader, sending a full tablet report...
I20260806 16:08:07.937084 8617 heartbeater.cc:503] Master 127.13.148.62:44871 was elected leader, sending a full tablet report...
I20260806 16:08:07.937500 8645 ts_tablet_manager.cc:1428] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 16:08:07.937655 8439 heartbeater.cc:503] Master 127.13.148.62:44871 was elected leader, sending a full tablet report...
I20260806 16:08:07.938200 8647 ts_tablet_manager.cc:1428] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260806 16:08:07.966023 8264 tablet.cc:2369] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:08:08.109216 8618 tablet.cc:2369] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:08:08.152354 8654 raft_consensus.cc:482] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:08:08.152570 8654 raft_consensus.cc:504] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:08.153811 8654 leader_election.cc:288] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bb9f10aa634458b94725db646a8399e (127.13.148.1:46707), 30b1368a51474598814ddaff16f63947 (127.13.148.2:41921)
I20260806 16:08:08.154841 8653 raft_consensus.cc:482] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:08:08.155061 8653 raft_consensus.cc:504] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:08.156276 8653 leader_election.cc:288] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8f1f1ab40cb4be296583921b9d7f2cd (127.13.148.3:40341), 30b1368a51474598814ddaff16f63947 (127.13.148.2:41921)
W20260806 16:08:08.157541 8440 tablet.cc:2369] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:08:08.158703 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.154231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:36347 (local address 127.13.148.1:46707)
0806 16:08:08.154381 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:08.154388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:08.154748 (+ 360us) server_negotiation.cc:407] Connection header received
0806 16:08:08.154961 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:08.154964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:08.155016 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:08.155096 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:08.155436 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.156129 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.157484 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.157958 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.158068 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.158124 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:08.158497 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 16:08:08.158569 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:08:08.159114 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.154122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46707 (local address 127.13.148.3:36347)
0806 16:08:08.154639 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:08.154651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:08.154828 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:08.155159 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:08.155162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:08.155183 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:08.155307 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:08.155315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.156266 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.156269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:08.157334 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:08.157347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.158239 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.158243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:08.158336 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.158359 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:08:08.158879 (+ 520us) client_negotiation.cc:240] Negotiation successful
0806 16:08:08.158959 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":118,"thread_start_us":119,"threads_started":1}
I20260806 16:08:08.159448 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.154119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41921 (local address 127.13.148.3:44701)
0806 16:08:08.154240 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:08.154258 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:08.154364 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:08.154762 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:08.154765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:08.154793 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:08.154946 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:08.154986 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.155959 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.155966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:08.157245 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:08.157256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.159008 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.159012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:08.159138 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.159169 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 16:08:08.159249 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 16:08:08.159322 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 16:08:08.159693 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df170da8110421abc2f02b386759e08" candidate_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb9f10aa634458b94725db646a8399e" is_pre_election: true
I20260806 16:08:08.159843 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.154227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:44701 (local address 127.13.148.2:41921)
0806 16:08:08.154360 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:08.154365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:08.154382 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:08:08.154538 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:08.154546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:08.154632 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:08.154723 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:08.155126 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.155832 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.158097 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.158779 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.158889 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.158941 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:08.159648 (+ 707us) server_negotiation.cc:299] Negotiation successful
0806 16:08:08.159726 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:08:08.159873 8200 raft_consensus.cc:2454] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f1f1ab40cb4be296583921b9d7f2cd in term 0.
I20260806 16:08:08.160439 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df170da8110421abc2f02b386759e08" candidate_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b1368a51474598814ddaff16f63947" is_pre_election: true
I20260806 16:08:08.160709 8378 raft_consensus.cc:2454] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f1f1ab40cb4be296583921b9d7f2cd in term 0.
I20260806 16:08:08.161219 8465 leader_election.cc:302] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30b1368a51474598814ddaff16f63947, 9bb9f10aa634458b94725db646a8399e, d8f1f1ab40cb4be296583921b9d7f2cd; no voters:
I20260806 16:08:08.161624 8654 raft_consensus.cc:2781] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:08:08.161754 8654 raft_consensus.cc:482] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:08:08.161830 8654 raft_consensus.cc:3037] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:08.163018 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.156965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:49155 (local address 127.13.148.2:41921)
0806 16:08:08.157485 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:08.157489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:08.158565 (+ 1076us) server_negotiation.cc:407] Connection header received
0806 16:08:08.158612 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:08.158615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:08.158659 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:08.158737 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:08.159135 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.159923 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.161542 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.162302 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.162443 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.162512 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:08.162796 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 16:08:08.162883 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":89,"threads_started":1}
I20260806 16:08:08.163043 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.157208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:58949 (local address 127.13.148.3:40341)
0806 16:08:08.157307 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:08.157311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:08.157325 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:08:08.157523 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:08.157526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:08.157579 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:08.157647 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:08.158057 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.158755 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.161790 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.162362 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.162483 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.162524 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:08.162881 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 16:08:08.162948 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 16:08:08.163035 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.156840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41921 (local address 127.13.148.1:49155)
0806 16:08:08.158347 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:08.158361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:08.158453 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:08.158857 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:08.158860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:08.158879 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:08.158991 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:08.158998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.160028 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.160035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:08.161417 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:08.161428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.162458 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.162484 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:08.162647 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.162675 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 16:08:08.162764 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 16:08:08.162847 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"mutex_wait_us":58,"thread_start_us":114,"threads_started":1}
I20260806 16:08:08.163398 8654 raft_consensus.cc:504] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:08.163545 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.156473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:40341 (local address 127.13.148.1:58949)
0806 16:08:08.157266 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:08.157282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:08.157392 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:08.157786 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:08.157789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:08.157809 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:08.157932 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:08.157937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.159041 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.159045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:08.161652 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:08.161664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.162585 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.162588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:08.162716 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.162759 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 16:08:08.163213 (+ 454us) client_negotiation.cc:240] Negotiation successful
0806 16:08:08.163290 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":338,"thread_start_us":84,"threads_started":1}
I20260806 16:08:08.163864 8654 leader_election.cc:288] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [CANDIDATE]: Term 1 election: Requested vote from peers 9bb9f10aa634458b94725db646a8399e (127.13.148.1:46707), 30b1368a51474598814ddaff16f63947 (127.13.148.2:41921)
I20260806 16:08:08.164059 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df170da8110421abc2f02b386759e08" candidate_uuid: "9bb9f10aa634458b94725db646a8399e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" is_pre_election: true
I20260806 16:08:08.164206 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df170da8110421abc2f02b386759e08" candidate_uuid: "9bb9f10aa634458b94725db646a8399e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b1368a51474598814ddaff16f63947" is_pre_election: true
I20260806 16:08:08.164317 8556 raft_consensus.cc:2379] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bb9f10aa634458b94725db646a8399e in current term 1: Already voted for candidate d8f1f1ab40cb4be296583921b9d7f2cd in this term.
I20260806 16:08:08.164306 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df170da8110421abc2f02b386759e08" candidate_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30b1368a51474598814ddaff16f63947"
I20260806 16:08:08.164345 8378 raft_consensus.cc:2454] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bb9f10aa634458b94725db646a8399e in term 0.
I20260806 16:08:08.164343 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df170da8110421abc2f02b386759e08" candidate_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb9f10aa634458b94725db646a8399e"
I20260806 16:08:08.164450 8200 raft_consensus.cc:3037] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:08.164773 8107 leader_election.cc:302] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30b1368a51474598814ddaff16f63947, 9bb9f10aa634458b94725db646a8399e; no voters:
I20260806 16:08:08.166415 8200 raft_consensus.cc:2454] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f1f1ab40cb4be296583921b9d7f2cd in term 1.
I20260806 16:08:08.166572 8653 raft_consensus.cc:2741] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260806 16:08:08.166826 8465 leader_election.cc:302] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bb9f10aa634458b94725db646a8399e, d8f1f1ab40cb4be296583921b9d7f2cd; no voters: 30b1368a51474598814ddaff16f63947
I20260806 16:08:08.167042 8654 raft_consensus.cc:2781] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:08.167297 8654 raft_consensus.cc:686] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 1 LEADER]: Becoming Leader. State: Replica: d8f1f1ab40cb4be296583921b9d7f2cd, State: Running, Role: LEADER
I20260806 16:08:08.167619 8654 consensus_queue.cc:237] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } }
I20260806 16:08:08.170971 8011 catalog_manager.cc:5495] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f1f1ab40cb4be296583921b9d7f2cd (127.13.148.3). New cstate: current_term: 1 leader_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f1f1ab40cb4be296583921b9d7f2cd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 40341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 } health_report { overall_health: UNKNOWN } } }
I20260806 16:08:08.213074 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.209395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.13.148.3:40341)
0806 16:08:08.209534 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:08.209538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:08.209553 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:08:08.209689 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:08.209692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:08.209754 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:08.209845 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:08.210071 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.210614 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.211440 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:08.211747 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:08.211852 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:08.212224 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:08.212615 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:08.212777 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:08.212873 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 16:08:08.212907 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:08:08.219466 8377 raft_consensus.cc:3037] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:08.220161 8200 raft_consensus.cc:1261] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 1 FOLLOWER]: Refusing update from remote peer d8f1f1ab40cb4be296583921b9d7f2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:08.220885 8662 consensus_queue.cc:1035] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb9f10aa634458b94725db646a8399e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:08.221093 8377 raft_consensus.cc:1261] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 1 FOLLOWER]: Refusing update from remote peer d8f1f1ab40cb4be296583921b9d7f2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:08.221648 8662 consensus_queue.cc:1035] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "30b1368a51474598814ddaff16f63947" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:08.229048 8665 mvcc.cc:205] Tried to move back new op lower bound from 7315589071739654144 to 7315589071537020928. Current Snapshot: MvccSnapshot[applied={T|T < 7315589071739654144}]
I20260806 16:08:08.230813 8667 mvcc.cc:205] Tried to move back new op lower bound from 7315589071739654144 to 7315589071537020928. Current Snapshot: MvccSnapshot[applied={T|T < 7315589071739654144}]
I20260806 16:08:08.231995 8666 mvcc.cc:205] Tried to move back new op lower bound from 7315589071739654144 to 7315589071537020928. Current Snapshot: MvccSnapshot[applied={T|T < 7315589071739654144}]
I20260806 16:08:08.240594 8008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:08.240831 8008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:08.243752 8008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 077b53f652614633a40b63217965e61e: Sending DeleteTablet for 3 replicas of tablet 8df170da8110421abc2f02b386759e08
I20260806 16:08:08.244741 8532 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df170da8110421abc2f02b386759e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:08 UTC) from {username='slave'} at 127.0.0.1:53140
I20260806 16:08:08.244727 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df170da8110421abc2f02b386759e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:08 UTC) from {username='slave'} at 127.0.0.1:60614
I20260806 16:08:08.244781 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df170da8110421abc2f02b386759e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:08 UTC) from {username='slave'} at 127.0.0.1:57280
I20260806 16:08:08.245373 8680 tablet_replica.cc:330] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: stopping tablet replica
I20260806 16:08:08.245759 8680 raft_consensus.cc:2229] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:08:08.245790 8681 tablet_replica.cc:330] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e: stopping tablet replica
I20260806 16:08:08.246101 8681 raft_consensus.cc:2229] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:08.246197 8680 raft_consensus.cc:2258] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:08.246443 8681 raft_consensus.cc:2258] T 8df170da8110421abc2f02b386759e08 P 9bb9f10aa634458b94725db646a8399e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:08.246563 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 8088
I20260806 16:08:08.246624 8679 tablet_replica.cc:330] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: stopping tablet replica
I20260806 16:08:08.246927 8679 raft_consensus.cc:2229] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:08.247222 8679 raft_consensus.cc:2258] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:08.253003 8679 ts_tablet_manager.cc:1905] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:08.255982 8679 ts_tablet_manager.cc:1918] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:08.256088 8679 log.cc:1196] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/8df170da8110421abc2f02b386759e08
I20260806 16:08:08.256434 8679 ts_tablet_manager.cc:1939] T 8df170da8110421abc2f02b386759e08 P 30b1368a51474598814ddaff16f63947: Deleting consensus metadata
I20260806 16:08:08.257853 7991 catalog_manager.cc:4841] TS 30b1368a51474598814ddaff16f63947 (127.13.148.2:41921): tablet 8df170da8110421abc2f02b386759e08 (table test-table [id=6b8d376bd67e4a31b75f1e2d773beafc]) successfully deleted
I20260806 16:08:08.265166 8680 ts_tablet_manager.cc:1905] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:08.267570 8680 ts_tablet_manager.cc:1918] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:08.267706 8680 log.cc:1196] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/8df170da8110421abc2f02b386759e08
I20260806 16:08:08.268000 8680 ts_tablet_manager.cc:1939] T 8df170da8110421abc2f02b386759e08 P d8f1f1ab40cb4be296583921b9d7f2cd: Deleting consensus metadata
I20260806 16:08:08.269233 7991 catalog_manager.cc:4841] TS d8f1f1ab40cb4be296583921b9d7f2cd (127.13.148.3:40341): tablet 8df170da8110421abc2f02b386759e08 (table test-table [id=6b8d376bd67e4a31b75f1e2d773beafc]) successfully deleted
W20260806 16:08:08.272863 7993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:46707 (error 108)
I20260806 16:08:08.273862 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 8266
I20260806 16:08:08.274034 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:08.273704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46707 (local address 127.0.0.1:60616)
0806 16:08:08.273863 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:08.273941 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:46707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 16:08:08.274225 7993 catalog_manager.cc:4584] TS 9bb9f10aa634458b94725db646a8399e (127.13.148.1:46707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8df170da8110421abc2f02b386759e08: Network error: Client connection negotiation failed: client connection to 127.13.148.1:46707: connect: Connection refused (error 111)
I20260806 16:08:08.288769 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 8443
I20260806 16:08:08.305859 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 7973
2026-08-06T16:08:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1303 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:08 dist-test-slave-7kzw krb5kdc[8685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:08 dist-test-slave-7kzw krb5kdc[8685](info): set up 2 sockets
Aug 06 16:08:08 dist-test-slave-7kzw krb5kdc[8685](info): commencing operation
krb5kdc: starting...
W20260806 16:08:09.529619 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:09 dist-test-slave-7kzw krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:09.665819 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45631
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:45631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:09.783448 8703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:09.783726 8703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:09.783767 8703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:09.787858 8703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:09.787918 8703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:09.787937 8703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:09.787953 8703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:09.787969 8703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:09.793220 8703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:45631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:09.794253 8703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:09.795336 8703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:09.804168 8714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:09.804168 8716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:09.804219 8711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:09.804572 8703 server_base.cc:1034] running on GCE node
I20260806 16:08:09.805231 8703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:09.806346 8703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:09.807550 8703 hybrid_clock.cc:648] HybridClock initialized: now 1786032489807494 us; error 69 us; skew 500 ppm
Aug 06 16:08:09 dist-test-slave-7kzw krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032489, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:09.810875 8703 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:09.811892 8703 webserver.cc:460] Webserver started at http://127.13.148.62:44617/ using document root <none> and password file <none>
I20260806 16:08:09.812461 8703 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:09.812529 8703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:09.812718 8703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:09.815011 8703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "7ec6a586224645eebfcd4a0bec07db29"
format_stamp: "Formatted at 2026-08-06 16:08:09 on dist-test-slave-7kzw"
server_key: "52c9f4ba00bfd22a33457c54327001d0"
server_key_iv: "594cb9b7c2115bf14ca664958931284b"
server_key_version: "encryptionkey@0"
I20260806 16:08:09.815613 8703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "7ec6a586224645eebfcd4a0bec07db29"
format_stamp: "Formatted at 2026-08-06 16:08:09 on dist-test-slave-7kzw"
server_key: "52c9f4ba00bfd22a33457c54327001d0"
server_key_iv: "594cb9b7c2115bf14ca664958931284b"
server_key_version: "encryptionkey@0"
I20260806 16:08:09.819583 8703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:09.822264 8724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:09.823652 8703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:08:09.823794 8703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "7ec6a586224645eebfcd4a0bec07db29"
format_stamp: "Formatted at 2026-08-06 16:08:09 on dist-test-slave-7kzw"
server_key: "52c9f4ba00bfd22a33457c54327001d0"
server_key_iv: "594cb9b7c2115bf14ca664958931284b"
server_key_version: "encryptionkey@0"
I20260806 16:08:09.823908 8703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:09.847522 8703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:09.850910 8703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:09.851126 8703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:09.858466 8703 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:45631
I20260806 16:08:09.858511 8799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:45631 every 8 connection(s)
I20260806 16:08:09.859494 8703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:09.862293 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:09.864375 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 8703
I20260806 16:08:09.864539 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:09.864845 13904 external_mini_cluster.cc:1428] Setting key 78e3de902a95f800196f567e185a2bfa
I20260806 16:08:09.868373 8800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29: Bootstrap starting.
I20260806 16:08:09.870667 8800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:09.871428 8800 log.cc:824] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:09.873301 8800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29: No bootstrap required, opened a new log
Aug 06 16:08:09 dist-test-slave-7kzw krb5kdc[8685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:09.876374 8800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6a586224645eebfcd4a0bec07db29" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45631 } }
I20260806 16:08:09.876639 8800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:09.876725 8800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec6a586224645eebfcd4a0bec07db29, State: Initialized, Role: FOLLOWER
I20260806 16:08:09.877203 8800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6a586224645eebfcd4a0bec07db29" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45631 } }
I20260806 16:08:09.877360 8800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:09.877435 8800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:09.877559 8800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:09.878808 8800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6a586224645eebfcd4a0bec07db29" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45631 } }
I20260806 16:08:09.879189 8800 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec6a586224645eebfcd4a0bec07db29; no voters:
I20260806 16:08:09.879545 8800 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:09.879700 8806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:09.879980 8806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec6a586224645eebfcd4a0bec07db29, State: Running, Role: LEADER
I20260806 16:08:09.879987 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:09.866260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.13.148.62:45631)
0806 16:08:09.866719 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:09.866735 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:09.866772 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:08:09.867521 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:09.867541 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:09.867875 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:09.868220 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:09.869063 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:09.870045 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:09.871016 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:09.871489 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:09.871664 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:09.874234 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:09.874239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:09.874253 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:09.874280 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:09.876610 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:09.877124 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:09.877136 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:09.877141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:09.877264 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:09.877584 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:09.877588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:09.877590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:09.878130 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:09.878351 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:09.878661 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 16:08:09.878982 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":155,"threads_started":1}
I20260806 16:08:09.880327 8806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6a586224645eebfcd4a0bec07db29" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45631 } }
I20260806 16:08:09.880842 8800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:09.882794 8807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ec6a586224645eebfcd4a0bec07db29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6a586224645eebfcd4a0bec07db29" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45631 } } }
I20260806 16:08:09.882921 8807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:09.883517 8808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec6a586224645eebfcd4a0bec07db29. Latest consensus state: current_term: 1 leader_uuid: "7ec6a586224645eebfcd4a0bec07db29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec6a586224645eebfcd4a0bec07db29" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45631 } } }
I20260806 16:08:09.883617 8822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:09.883622 8808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:09.886962 8822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:09.892736 8822 catalog_manager.cc:1347] Generated new cluster ID: ed4b01f72dad463cba8611c361fccb06
I20260806 16:08:09.892810 8822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:09.911289 8822 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:09.912405 8822 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:09.920933 8822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29: Generated new TSK 0
I20260806 16:08:09.921818 8822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260806 16:08:09.961925 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45631
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:10.075883 8833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:10.076190 8833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:10.076269 8833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:10.080328 8833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:10.080416 8833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:10.080500 8833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:08:10.086086 8833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:10.087203 8833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:10.088284 8833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:10.098424 8844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:10.098506 8833 server_base.cc:1034] running on GCE node
W20260806 16:08:10.098444 8841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:10.098443 8840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:10.099097 8833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:10.099720 8833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:10.100888 8833 hybrid_clock.cc:648] HybridClock initialized: now 1786032490100873 us; error 32 us; skew 500 ppm
Aug 06 16:08:10 dist-test-slave-7kzw krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032490, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:10.104065 8833 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260806 16:08:10.104996 8833 webserver.cc:460] Webserver started at http://127.13.148.1:37391/ using document root <none> and password file <none>
I20260806 16:08:10.105581 8833 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:10.105674 8833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:10.105911 8833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:10.108026 8833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "cc73ff64c4f945bcbaac53b101a25582"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "caf7b99ef900efddd7eca39b4edc82f8"
server_key_iv: "181807302b6c824e988a48cbbe8235b0"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.108606 8833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "cc73ff64c4f945bcbaac53b101a25582"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "caf7b99ef900efddd7eca39b4edc82f8"
server_key_iv: "181807302b6c824e988a48cbbe8235b0"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.112448 8833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:08:10.114924 8852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.116186 8833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:10.116339 8833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "cc73ff64c4f945bcbaac53b101a25582"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "caf7b99ef900efddd7eca39b4edc82f8"
server_key_iv: "181807302b6c824e988a48cbbe8235b0"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.116463 8833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:10.134065 8833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:10.137198 8833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:10.137405 8833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:10.138022 8833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:10.138938 8833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:10.139010 8833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.139082 8833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:10.139124 8833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.148401 8833 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:35361
I20260806 16:08:10.148427 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:35361 every 8 connection(s)
I20260806 16:08:10.149305 8833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:08:10.149887 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 8833
I20260806 16:08:10.150038 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:08:10.150318 13904 external_mini_cluster.cc:1428] Setting key e0dd93b4d32ac5f7fdc689b164f6a8d2
Aug 06 16:08:10 dist-test-slave-7kzw krb5kdc[8685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032490, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260806 16:08:10.166528 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.151395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38203 (local address 127.13.148.62:45631)
0806 16:08:10.151532 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.151536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.152624 (+ 1088us) server_negotiation.cc:407] Connection header received
0806 16:08:10.155013 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.155019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.155087 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.155214 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:10.156474 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.157367 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.158757 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.159215 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.159365 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.162285 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:10.162292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:10.162299 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:10.162339 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:10.164163 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:10.164733 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:10.164738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:10.164740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:10.164844 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:10.165238 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:10.165242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:10.165244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:10.165456 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:10.165593 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.166127 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.166275 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:08:10.167354 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.151853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45631 (local address 127.13.148.1:38203)
0806 16:08:10.152439 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.152501 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:10.154726 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:10.155366 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:10.155373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:10.155767 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:10.156303 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.156318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.157529 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.157532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.158624 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.158635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.159410 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.159413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.159568 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.160278 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:10.160295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:10.162089 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:10.164295 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:10.164302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:10.164319 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:10.164616 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:10.164956 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:10.164959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:10.164961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:10.165124 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:10.165593 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:10.165601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:10.165865 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 16:08:10.166081 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 16:08:10.166430 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":146,"threads_started":1}
I20260806 16:08:10.168705 9011 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45631
I20260806 16:08:10.169000 9011 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:10.169663 9011 heartbeater.cc:511] Master 127.13.148.62:45631 requested a full tablet report, sending...
I20260806 16:08:10.171531 8745 ts_manager.cc:194] Registered new tserver with Master: cc73ff64c4f945bcbaac53b101a25582 (127.13.148.1:35361)
I20260806 16:08:10.173306 8745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:38203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260806 16:08:10.185586 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45631
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:10.305892 9019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:10.306174 9019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:10.306255 9019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:10.310400 9019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:10.310493 9019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:10.310626 9019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:08:10.316143 9019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:10.317281 9019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:10.318357 9019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:10.328560 9032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:10.328653 9028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:10.328550 9030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:10.328836 9019 server_base.cc:1034] running on GCE node
I20260806 16:08:10.329309 9019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:10.330015 9019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:10.331174 9019 hybrid_clock.cc:648] HybridClock initialized: now 1786032490331141 us; error 44 us; skew 500 ppm
Aug 06 16:08:10 dist-test-slave-7kzw krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032490, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:10.334080 9019 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260806 16:08:10.335054 9019 webserver.cc:460] Webserver started at http://127.13.148.2:45003/ using document root <none> and password file <none>
I20260806 16:08:10.335626 9019 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:10.335754 9019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:10.335986 9019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:10.338035 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "384d20da5d5f49cd9d0d0630f14e4b5c"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "908fa427f11cb749dbbcc8e9d57cc879"
server_key_iv: "5312b53f62e80c56a6ebe5c9c48f01b1"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.338701 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "384d20da5d5f49cd9d0d0630f14e4b5c"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "908fa427f11cb749dbbcc8e9d57cc879"
server_key_iv: "5312b53f62e80c56a6ebe5c9c48f01b1"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.342307 9019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:08:10.344798 9040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.345960 9019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:10.346093 9019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "384d20da5d5f49cd9d0d0630f14e4b5c"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "908fa427f11cb749dbbcc8e9d57cc879"
server_key_iv: "5312b53f62e80c56a6ebe5c9c48f01b1"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.346199 9019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:10.360278 9019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:10.363420 9019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:10.363622 9019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:10.364223 9019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:10.365036 9019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:10.365105 9019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.365176 9019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:10.365222 9019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.374169 9019 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:46183
I20260806 16:08:10.374198 9200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:46183 every 8 connection(s)
I20260806 16:08:10.375108 9019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:08:10.383486 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 9019
I20260806 16:08:10.383654 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:08:10.383976 13904 external_mini_cluster.cc:1428] Setting key baa58e0ddb369d63f196e2c3ff56e253
Aug 06 16:08:10 dist-test-slave-7kzw krb5kdc[8685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032490, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:10.389252 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.376880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:37439 (local address 127.13.148.62:45631)
0806 16:08:10.377027 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.377031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.377834 (+ 803us) server_negotiation.cc:407] Connection header received
0806 16:08:10.379346 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.379349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.379397 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.379473 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:10.380561 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.381291 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.382620 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.382897 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.382979 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.385672 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:10.385676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:10.385678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:10.385702 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:10.387214 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:10.387742 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:10.387747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:10.387749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:10.387831 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:10.388162 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:10.388166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:10.388168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:10.388347 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:10.388452 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.388942 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.389080 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:08:10.390028 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.377232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45631 (local address 127.13.148.2:37439)
0806 16:08:10.377686 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.377728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:10.379087 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:10.379605 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:10.379612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:10.379954 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:10.380407 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.380419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.381443 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.381446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.382457 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.382502 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.383029 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.383032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.383176 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.383927 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:10.383943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:10.385497 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:10.387348 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:10.387353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:10.387367 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:10.387637 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:10.387921 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:10.387925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:10.387927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:10.388068 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:10.388451 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:10.388457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:10.388703 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 16:08:10.388884 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 16:08:10.389203 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260806 16:08:10.391348 9201 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45631
I20260806 16:08:10.391621 9201 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:10.392112 9201 heartbeater.cc:511] Master 127.13.148.62:45631 requested a full tablet report, sending...
I20260806 16:08:10.393289 8745 ts_manager.cc:194] Registered new tserver with Master: 384d20da5d5f49cd9d0d0630f14e4b5c (127.13.148.2:46183)
I20260806 16:08:10.394119 8745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:37439
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260806 16:08:10.413582 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45631
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:10.532054 9210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:10.532346 9210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:10.532441 9210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:10.536504 9210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:10.536588 9210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:10.536679 9210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:08:10.541996 9210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:10.543146 9210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:10.544178 9210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:10.554111 9218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:10.554085 9220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:10.554131 9210 server_base.cc:1034] running on GCE node
W20260806 16:08:10.554081 9222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:10.554826 9210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:10.555446 9210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:10.556605 9210 hybrid_clock.cc:648] HybridClock initialized: now 1786032490556582 us; error 38 us; skew 500 ppm
Aug 06 16:08:10 dist-test-slave-7kzw krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032490, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:10.559404 9210 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260806 16:08:10.560415 9210 webserver.cc:460] Webserver started at http://127.13.148.3:37827/ using document root <none> and password file <none>
I20260806 16:08:10.560988 9210 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:10.561055 9210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:10.561268 9210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:10.563340 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "083a0557af26431fb32cac22ccaf756a"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "76ce1dc791e59b87af96cd06cf73f95c"
server_key_iv: "ac006c256eaa36765a5ad05a7c7499b2"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.563897 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "083a0557af26431fb32cac22ccaf756a"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "76ce1dc791e59b87af96cd06cf73f95c"
server_key_iv: "ac006c256eaa36765a5ad05a7c7499b2"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.567590 9210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:08:10.570087 9232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.571497 9210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:08:10.571712 9210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "083a0557af26431fb32cac22ccaf756a"
format_stamp: "Formatted at 2026-08-06 16:08:10 on dist-test-slave-7kzw"
server_key: "76ce1dc791e59b87af96cd06cf73f95c"
server_key_iv: "ac006c256eaa36765a5ad05a7c7499b2"
server_key_version: "encryptionkey@0"
I20260806 16:08:10.571843 9210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:10.599575 9210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:10.603044 9210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:10.603266 9210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:10.604027 9210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:10.604947 9210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:10.605026 9210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.605103 9210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:10.605157 9210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:10.614995 9210 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:45877
I20260806 16:08:10.615015 9383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:45877 every 8 connection(s)
I20260806 16:08:10.615988 9210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:08:10.621309 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 9210
I20260806 16:08:10.621487 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:08:10.621834 13904 external_mini_cluster.cc:1428] Setting key 5ce437edbbcfb1ad85bce72ce559d376
Aug 06 16:08:10 dist-test-slave-7kzw krb5kdc[8685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032490, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:10.631495 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.617964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:41167 (local address 127.13.148.62:45631)
0806 16:08:10.618130 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.618136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.619184 (+ 1048us) server_negotiation.cc:407] Connection header received
0806 16:08:10.620589 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.620592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.620645 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.620726 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:10.622075 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.622916 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.624472 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.624714 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.624803 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.627623 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:10.627626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:10.627629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:10.627655 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:10.629329 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:10.629880 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:10.629883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:10.629884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:10.629966 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:10.630298 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:10.630300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:10.630302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:10.630550 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:10.630655 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.631233 (+ 578us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.631350 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:08:10.632321 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.618515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45631 (local address 127.13.148.3:41167)
0806 16:08:10.619029 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.619075 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:10.620285 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:10.620905 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:10.620912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:10.621300 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:10.621888 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.621903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.623032 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.623035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.624354 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.624371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.624854 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.624857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.624996 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.625590 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:10.625606 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:10.627453 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:10.629459 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:10.629464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:10.629483 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:10.629783 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:10.630056 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:10.630058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:10.630060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:10.630209 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:10.630707 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:10.630713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:10.630954 (+ 241us) client_negotiation.cc:769] Sending connection context
0806 16:08:10.631144 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 16:08:10.631416 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":124,"threads_started":1}
I20260806 16:08:10.633755 9386 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45631
I20260806 16:08:10.634037 9386 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:10.634632 9386 heartbeater.cc:511] Master 127.13.148.62:45631 requested a full tablet report, sending...
I20260806 16:08:10.635866 8745 ts_manager.cc:194] Registered new tserver with Master: 083a0557af26431fb32cac22ccaf756a (127.13.148.3:45877)
I20260806 16:08:10.636508 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:08:10.636709 8745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:41167
I20260806 16:08:10.647068 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.638816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.13.148.62:45631)
0806 16:08:10.638941 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.638945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.639210 (+ 265us) server_negotiation.cc:407] Connection header received
0806 16:08:10.639355 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.639358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.639406 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.639491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:10.640280 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.640832 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.641768 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.642009 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.642137 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.643733 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:10.643736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:10.643738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:10.643764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:10.645258 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:10.645736 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:10.645740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:10.645746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:10.645827 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:10.646104 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:10.646108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:10.646110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:10.646332 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:10.646476 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.646779 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.646902 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:08:10.650027 8745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:08:10.653128 8745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:08:10.667786 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.661591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46183 (local address 127.0.0.1:45356)
0806 16:08:10.662001 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.662017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:10.662174 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:10.663636 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:10.663639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:10.663657 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:10.663756 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.663762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.665000 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.665004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.666352 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.666364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.667243 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.667247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.667333 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.667368 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 16:08:10.667415 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 16:08:10.667561 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":91,"spinlock_wait_cycles":103168,"thread_start_us":114,"threads_started":1}
I20260806 16:08:10.668140 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.661441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:35361 (local address 127.0.0.1:39798)
0806 16:08:10.661958 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.662007 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:10.662162 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:10.663046 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:10.663052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:10.663092 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:10.663251 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.663262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.664728 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.664732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.666042 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.666053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.667085 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.667089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.667222 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.667268 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 16:08:10.667386 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 16:08:10.667561 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":63,"thread_start_us":95,"threads_started":1}
I20260806 16:08:10.668172 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.661595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.13.148.1:35361)
0806 16:08:10.662266 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.662286 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.662312 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:08:10.662391 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.662399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.662639 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.662944 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:10.663376 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.664588 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.666186 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.666950 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.667064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.667226 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.667650 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.667761 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":103,"threads_started":1}
I20260806 16:08:10.668257 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.662707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.13.148.2:46183)
0806 16:08:10.663125 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.663137 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.663163 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:08:10.663243 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.663248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.663454 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.663648 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:10.663856 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.664880 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.666532 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.667100 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.667659 (+ 559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.667788 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.667836 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.667922 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":113,"threads_started":1}
I20260806 16:08:10.667814 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.662202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45877 (local address 127.0.0.1:48326)
0806 16:08:10.663139 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.663154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:10.663438 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:10.663923 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:10.663925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:10.663942 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:10.664019 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.664025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.665050 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.665053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.666352 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.666364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.667426 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.667430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.667514 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.667537 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:08:10.667629 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 16:08:10.667706 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":96,"thread_start_us":125,"threads_started":1}
I20260806 16:08:10.668634 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.662545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.13.148.3:45877)
0806 16:08:10.662960 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.662971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.663395 (+ 424us) server_negotiation.cc:407] Connection header received
0806 16:08:10.663588 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.663607 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.663800 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.663939 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:10.664118 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.664939 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.666506 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.667307 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.667422 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.667551 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.667872 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.668136 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":152,"threads_started":1}
I20260806 16:08:10.671341 9304 tablet_service.cc:1467] Processing CreateTablet for tablet bd1308e4e1e14ea7967662842887f22f (DEFAULT_TABLE table=test-table [id=e66e18d576d24552a97b8b05130d3113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:10.671334 8922 tablet_service.cc:1467] Processing CreateTablet for tablet bd1308e4e1e14ea7967662842887f22f (DEFAULT_TABLE table=test-table [id=e66e18d576d24552a97b8b05130d3113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:10.671336 9110 tablet_service.cc:1467] Processing CreateTablet for tablet bd1308e4e1e14ea7967662842887f22f (DEFAULT_TABLE table=test-table [id=e66e18d576d24552a97b8b05130d3113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:10.672348 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1308e4e1e14ea7967662842887f22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:10.672353 9110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1308e4e1e14ea7967662842887f22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:10.672353 8922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1308e4e1e14ea7967662842887f22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:10.678821 9410 tablet_bootstrap.cc:492] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: Bootstrap starting.
I20260806 16:08:10.680017 9412 tablet_bootstrap.cc:492] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: Bootstrap starting.
I20260806 16:08:10.680145 9411 tablet_bootstrap.cc:492] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582: Bootstrap starting.
I20260806 16:08:10.681486 9410 tablet_bootstrap.cc:654] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:10.682175 9411 tablet_bootstrap.cc:654] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:10.682170 9412 tablet_bootstrap.cc:654] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:10.683014 9410 log.cc:824] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:10.683145 9411 log.cc:824] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:10.683491 9412 log.cc:824] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:10.685009 9410 tablet_bootstrap.cc:492] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: No bootstrap required, opened a new log
I20260806 16:08:10.685106 9411 tablet_bootstrap.cc:492] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582: No bootstrap required, opened a new log
I20260806 16:08:10.685228 9410 ts_tablet_manager.cc:1397] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 16:08:10.685304 9411 ts_tablet_manager.cc:1397] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260806 16:08:10.685367 9412 tablet_bootstrap.cc:492] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: No bootstrap required, opened a new log
I20260806 16:08:10.685534 9412 ts_tablet_manager.cc:1397] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 16:08:10.688889 9410 raft_consensus.cc:348] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.689164 9410 raft_consensus.cc:374] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:10.689221 9410 raft_consensus.cc:729] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083a0557af26431fb32cac22ccaf756a, State: Initialized, Role: FOLLOWER
I20260806 16:08:10.689667 9410 consensus_queue.cc:260] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.689733 9411 raft_consensus.cc:348] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.689743 9412 raft_consensus.cc:348] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.689987 9412 raft_consensus.cc:374] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:10.689987 9411 raft_consensus.cc:374] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:10.690053 9412 raft_consensus.cc:729] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384d20da5d5f49cd9d0d0630f14e4b5c, State: Initialized, Role: FOLLOWER
I20260806 16:08:10.690063 9411 raft_consensus.cc:729] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc73ff64c4f945bcbaac53b101a25582, State: Initialized, Role: FOLLOWER
I20260806 16:08:10.690549 9386 heartbeater.cc:503] Master 127.13.148.62:45631 was elected leader, sending a full tablet report...
I20260806 16:08:10.690529 9412 consensus_queue.cc:260] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.690544 9411 consensus_queue.cc:260] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.690757 9410 ts_tablet_manager.cc:1428] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260806 16:08:10.691442 9412 ts_tablet_manager.cc:1428] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 16:08:10.691442 9411 ts_tablet_manager.cc:1428] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 16:08:10.691821 9011 heartbeater.cc:503] Master 127.13.148.62:45631 was elected leader, sending a full tablet report...
I20260806 16:08:10.692736 9201 heartbeater.cc:503] Master 127.13.148.62:45631 was elected leader, sending a full tablet report...
I20260806 16:08:10.753121 9417 raft_consensus.cc:482] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:08:10.753402 9417 raft_consensus.cc:504] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.754629 9417 leader_election.cc:288] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 384d20da5d5f49cd9d0d0630f14e4b5c (127.13.148.2:46183), 083a0557af26431fb32cac22ccaf756a (127.13.148.3:45877)
I20260806 16:08:10.759186 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.755086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:34615 (local address 127.13.148.2:46183)
0806 16:08:10.755239 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.755243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.755548 (+ 305us) server_negotiation.cc:407] Connection header received
0806 16:08:10.755713 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.755720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.755786 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.755911 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:10.756356 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.756881 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.758070 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.758572 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.758708 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.758771 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.758964 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.759045 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:08:10.759202 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.755049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46183 (local address 127.13.148.1:34615)
0806 16:08:10.755409 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.755429 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:10.755576 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:10.756076 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:10.756079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:10.756097 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:10.756219 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.756225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.757004 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.757009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.757917 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.757927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.758701 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.758705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.758810 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.758841 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 16:08:10.758945 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:08:10.759031 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":182,"threads_started":1}
I20260806 16:08:10.759967 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.755657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45877 (local address 127.13.148.1:51361)
0806 16:08:10.756055 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.756068 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:10.756196 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:10.756474 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:10.756476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:10.756494 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:10.756635 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.756641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.757736 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.757743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.758626 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:10.758635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.759514 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.759518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:10.759631 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.759655 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 16:08:10.759736 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 16:08:10.759803 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260806 16:08:10.759996 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.755845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:51361 (local address 127.13.148.3:45877)
0806 16:08:10.756011 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.756015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.756153 (+ 138us) server_negotiation.cc:407] Connection header received
0806 16:08:10.756317 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.756320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.756373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.756461 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:10.756751 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.757625 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.758763 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.759229 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.759391 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.759450 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.759792 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.759869 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 16:08:10.759990 9142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1308e4e1e14ea7967662842887f22f" candidate_uuid: "cc73ff64c4f945bcbaac53b101a25582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" is_pre_election: true
I20260806 16:08:10.760341 9142 raft_consensus.cc:2454] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc73ff64c4f945bcbaac53b101a25582 in term 0.
I20260806 16:08:10.760771 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1308e4e1e14ea7967662842887f22f" candidate_uuid: "cc73ff64c4f945bcbaac53b101a25582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083a0557af26431fb32cac22ccaf756a" is_pre_election: true
I20260806 16:08:10.760972 8857 leader_election.cc:302] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384d20da5d5f49cd9d0d0630f14e4b5c, cc73ff64c4f945bcbaac53b101a25582; no voters:
I20260806 16:08:10.761108 9329 raft_consensus.cc:2454] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc73ff64c4f945bcbaac53b101a25582 in term 0.
I20260806 16:08:10.761327 9417 raft_consensus.cc:2781] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:08:10.761439 9417 raft_consensus.cc:482] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:08:10.761505 9417 raft_consensus.cc:3037] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:10.762741 9417 raft_consensus.cc:504] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.763125 9417 leader_election.cc:288] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [CANDIDATE]: Term 1 election: Requested vote from peers 384d20da5d5f49cd9d0d0630f14e4b5c (127.13.148.2:46183), 083a0557af26431fb32cac22ccaf756a (127.13.148.3:45877)
I20260806 16:08:10.763549 9142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1308e4e1e14ea7967662842887f22f" candidate_uuid: "cc73ff64c4f945bcbaac53b101a25582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c"
I20260806 16:08:10.763558 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1308e4e1e14ea7967662842887f22f" candidate_uuid: "cc73ff64c4f945bcbaac53b101a25582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083a0557af26431fb32cac22ccaf756a"
I20260806 16:08:10.763703 9142 raft_consensus.cc:3037] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:10.763700 9329 raft_consensus.cc:3037] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:10.764801 9142 raft_consensus.cc:2454] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc73ff64c4f945bcbaac53b101a25582 in term 1.
I20260806 16:08:10.764855 9329 raft_consensus.cc:2454] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc73ff64c4f945bcbaac53b101a25582 in term 1.
I20260806 16:08:10.765164 8857 leader_election.cc:302] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384d20da5d5f49cd9d0d0630f14e4b5c, cc73ff64c4f945bcbaac53b101a25582; no voters:
I20260806 16:08:10.765398 9417 raft_consensus.cc:2781] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:10.765656 9417 raft_consensus.cc:686] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [term 1 LEADER]: Becoming Leader. State: Replica: cc73ff64c4f945bcbaac53b101a25582, State: Running, Role: LEADER
I20260806 16:08:10.766027 9417 consensus_queue.cc:237] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } }
I20260806 16:08:10.769691 8744 catalog_manager.cc:5495] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc73ff64c4f945bcbaac53b101a25582 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "cc73ff64c4f945bcbaac53b101a25582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc73ff64c4f945bcbaac53b101a25582" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 35361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 } health_report { overall_health: UNKNOWN } } }
I20260806 16:08:10.794237 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.790855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.13.148.1:35361)
0806 16:08:10.790988 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:10.790992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:10.791006 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:08:10.791101 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:10.791104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:10.791164 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:10.791256 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:10.791478 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.791991 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.792861 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:10.793074 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:10.793171 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:10.793422 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:10.793852 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:10.793948 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:10.794056 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 16:08:10.794105 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:08:10.800776 9329 raft_consensus.cc:1261] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 1 FOLLOWER]: Refusing update from remote peer cc73ff64c4f945bcbaac53b101a25582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:10.800776 9142 raft_consensus.cc:1261] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 1 FOLLOWER]: Refusing update from remote peer cc73ff64c4f945bcbaac53b101a25582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:10.801694 9417 consensus_queue.cc:1035] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083a0557af26431fb32cac22ccaf756a" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:10.801960 9423 consensus_queue.cc:1035] T bd1308e4e1e14ea7967662842887f22f P cc73ff64c4f945bcbaac53b101a25582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "384d20da5d5f49cd9d0d0630f14e4b5c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:10.811125 9428 mvcc.cc:205] Tried to move back new op lower bound from 7315589082311151616 to 7315589082180902912. Current Snapshot: MvccSnapshot[applied={T|T < 7315589082311151616}]
I20260806 16:08:10.811318 9429 mvcc.cc:205] Tried to move back new op lower bound from 7315589082311151616 to 7315589082180902912. Current Snapshot: MvccSnapshot[applied={T|T < 7315589082311151616}]
I20260806 16:08:10.819448 8742 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:10.819604 8742 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:10.822292 8742 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ec6a586224645eebfcd4a0bec07db29: Sending DeleteTablet for 3 replicas of tablet bd1308e4e1e14ea7967662842887f22f
I20260806 16:08:10.823081 8922 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1308e4e1e14ea7967662842887f22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:10 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:39798
I20260806 16:08:10.823287 9304 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1308e4e1e14ea7967662842887f22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:10 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:48326
I20260806 16:08:10.823326 9110 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1308e4e1e14ea7967662842887f22f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:10 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:45356
I20260806 16:08:10.823807 9445 tablet_replica.cc:330] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: stopping tablet replica
I20260806 16:08:10.823907 9446 tablet_replica.cc:330] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: stopping tablet replica
I20260806 16:08:10.824231 9446 raft_consensus.cc:2229] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:10.824250 9445 raft_consensus.cc:2229] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:10.824376 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 8833
I20260806 16:08:10.824570 9446 raft_consensus.cc:2258] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:10.824750 9445 raft_consensus.cc:2258] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:10.828074 9445 ts_tablet_manager.cc:1905] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:10.831432 9445 ts_tablet_manager.cc:1918] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:10.831570 9445 log.cc:1196] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/bd1308e4e1e14ea7967662842887f22f
I20260806 16:08:10.831894 9446 ts_tablet_manager.cc:1905] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:10.831935 9445 ts_tablet_manager.cc:1939] T bd1308e4e1e14ea7967662842887f22f P 083a0557af26431fb32cac22ccaf756a: Deleting consensus metadata
I20260806 16:08:10.833199 8728 catalog_manager.cc:4841] TS 083a0557af26431fb32cac22ccaf756a (127.13.148.3:45877): tablet bd1308e4e1e14ea7967662842887f22f (table test-table [id=e66e18d576d24552a97b8b05130d3113]) successfully deleted
I20260806 16:08:10.835557 9446 ts_tablet_manager.cc:1918] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:10.835672 9446 log.cc:1196] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/bd1308e4e1e14ea7967662842887f22f
I20260806 16:08:10.836054 9446 ts_tablet_manager.cc:1939] T bd1308e4e1e14ea7967662842887f22f P 384d20da5d5f49cd9d0d0630f14e4b5c: Deleting consensus metadata
I20260806 16:08:10.837471 8728 catalog_manager.cc:4841] TS 384d20da5d5f49cd9d0d0630f14e4b5c (127.13.148.2:46183): tablet bd1308e4e1e14ea7967662842887f22f (table test-table [id=e66e18d576d24552a97b8b05130d3113]) successfully deleted
W20260806 16:08:10.838061 8725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:35361 (error 108)
I20260806 16:08:10.838311 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 9019
I20260806 16:08:10.840728 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:10.839301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:35361 (local address 127.0.0.1:39816)
0806 16:08:10.840510 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:10.840611 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1086}
W20260806 16:08:10.840961 8725 catalog_manager.cc:4584] TS cc73ff64c4f945bcbaac53b101a25582 (127.13.148.1:35361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd1308e4e1e14ea7967662842887f22f: Network error: Client connection negotiation failed: client connection to 127.13.148.1:35361: connect: Connection refused (error 111)
I20260806 16:08:10.854230 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 9210
I20260806 16:08:10.864260 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 8703
2026-08-06T16:08:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2558 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T16:08:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:10Z Disabled control of system clock
I20260806 16:08:10.901284 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40999
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:08:11.016549 9454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:11.016952 9454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:11.016999 9454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:11.021343 9454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:11.021436 9454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:11.021456 9454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:11.021476 9454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:11.021493 9454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:11.027521 9454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:11.028676 9454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:11.030089 9454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:11.039697 9466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:11.039762 9462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:11.039700 9461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:11.040045 9454 server_base.cc:1034] running on GCE node
I20260806 16:08:11.040699 9454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:11.041886 9454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:11.043115 9454 hybrid_clock.cc:648] HybridClock initialized: now 1786032491043094 us; error 34 us; skew 500 ppm
I20260806 16:08:11.049855 9454 webserver.cc:460] Webserver started at http://127.13.148.62:34379/ using document root <none> and password file <none>
I20260806 16:08:11.050597 9454 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:11.050701 9454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:11.050961 9454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:11.053098 9454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "b67dbc124b9b4d588ffc9e011078dd70"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "6b53a98c8d624a61d9a317bedb0b36b1"
server_key_iv: "fcff7a368313f46401780e67bbebfc27"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.053675 9454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "b67dbc124b9b4d588ffc9e011078dd70"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "6b53a98c8d624a61d9a317bedb0b36b1"
server_key_iv: "fcff7a368313f46401780e67bbebfc27"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.057711 9454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:08:11.060658 9476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.061998 9454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 16:08:11.062275 9454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "b67dbc124b9b4d588ffc9e011078dd70"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "6b53a98c8d624a61d9a317bedb0b36b1"
server_key_iv: "fcff7a368313f46401780e67bbebfc27"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.062407 9454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:11.086449 9454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:11.087325 9454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:11.087553 9454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:11.095363 9454 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40999
I20260806 16:08:11.095381 9557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40999 every 8 connection(s)
I20260806 16:08:11.096385 9454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:11.098515 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 9454
I20260806 16:08:11.098662 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:11.098970 13904 external_mini_cluster.cc:1428] Setting key 417983a6a748604bf3893d94f1211c9b
I20260806 16:08:11.099942 9559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:11.107071 9559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70: Bootstrap starting.
I20260806 16:08:11.109393 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.100618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.13.148.62:40999)
0806 16:08:11.101317 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:11.101335 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:11.101375 (+ 40us) server_negotiation.cc:407] Connection header received
0806 16:08:11.102120 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:11.102139 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:11.102636 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:11.102952 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:11.103334 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.104440 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.105699 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.106264 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.106405 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.107396 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:11.107402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:11.107416 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:11.107880 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:11.107941 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:11.107964 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:11.108051 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:11.108235 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 16:08:11.108594 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":177,"threads_started":1}
I20260806 16:08:11.109633 9559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:11.110395 9559 log.cc:824] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:11.112479 9559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70: No bootstrap required, opened a new log
I20260806 16:08:11.116439 9559 raft_consensus.cc:348] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b67dbc124b9b4d588ffc9e011078dd70" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40999 } }
I20260806 16:08:11.116729 9559 raft_consensus.cc:374] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:11.116812 9559 raft_consensus.cc:729] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67dbc124b9b4d588ffc9e011078dd70, State: Initialized, Role: FOLLOWER
I20260806 16:08:11.117286 9559 consensus_queue.cc:260] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b67dbc124b9b4d588ffc9e011078dd70" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40999 } }
I20260806 16:08:11.117462 9559 raft_consensus.cc:388] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:11.117573 9559 raft_consensus.cc:482] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:11.117702 9559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:11.118883 9559 raft_consensus.cc:504] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b67dbc124b9b4d588ffc9e011078dd70" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40999 } }
I20260806 16:08:11.119215 9559 leader_election.cc:302] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b67dbc124b9b4d588ffc9e011078dd70; no voters:
I20260806 16:08:11.119498 9559 leader_election.cc:288] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:11.119721 9564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:11.119961 9564 raft_consensus.cc:686] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [term 1 LEADER]: Becoming Leader. State: Replica: b67dbc124b9b4d588ffc9e011078dd70, State: Running, Role: LEADER
I20260806 16:08:11.120312 9564 consensus_queue.cc:237] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b67dbc124b9b4d588ffc9e011078dd70" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40999 } }
I20260806 16:08:11.120903 9559 sys_catalog.cc:564] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:11.122139 9566 sys_catalog.cc:455] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b67dbc124b9b4d588ffc9e011078dd70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b67dbc124b9b4d588ffc9e011078dd70" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40999 } } }
I20260806 16:08:11.122212 9567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b67dbc124b9b4d588ffc9e011078dd70. Latest consensus state: current_term: 1 leader_uuid: "b67dbc124b9b4d588ffc9e011078dd70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b67dbc124b9b4d588ffc9e011078dd70" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40999 } } }
I20260806 16:08:11.122273 9566 sys_catalog.cc:458] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:11.122280 9567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:11.122786 9575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:11.125650 9575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:11.131474 9575 catalog_manager.cc:1347] Generated new cluster ID: 1795f6fa906a40bfb0daf8d42a8223b7
I20260806 16:08:11.131551 9575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:11.157105 9575 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:11.158054 9575 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:11.164595 9575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70: Generated new TSK 0
I20260806 16:08:11.165295 9575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:11.177783 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40999
--builtin_ntp_servers=127.13.148.20:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 16:08:11.293159 9590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:11.293417 9590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:11.293455 9590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:11.297469 9590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:11.297533 9590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:11.297586 9590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260806 16:08:11.302210 9590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:11.303316 9590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:11.304749 9590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:11.314723 9597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:11.314733 9596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:11.314778 9590 server_base.cc:1034] running on GCE node
W20260806 16:08:11.314733 9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:11.315464 9590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:11.316111 9590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:11.317330 9590 hybrid_clock.cc:648] HybridClock initialized: now 1786032491317291 us; error 51 us; skew 500 ppm
I20260806 16:08:11.319402 9590 webserver.cc:460] Webserver started at http://127.13.148.1:40431/ using document root <none> and password file <none>
I20260806 16:08:11.320039 9590 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:11.320142 9590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:11.320405 9590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:11.322844 9590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/data/instance:
uuid: "f3ae8791d2a74098b7871c1cab01e34e"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "39f3c24ab3adf9274918339197fc9741"
server_key_iv: "0c4e697c39b031c47ae4f04eab30b4b7"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.323498 9590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "f3ae8791d2a74098b7871c1cab01e34e"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "39f3c24ab3adf9274918339197fc9741"
server_key_iv: "0c4e697c39b031c47ae4f04eab30b4b7"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.327495 9590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:11.330209 9608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.331538 9590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:08:11.331693 9590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/wal
uuid: "f3ae8791d2a74098b7871c1cab01e34e"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "39f3c24ab3adf9274918339197fc9741"
server_key_iv: "0c4e697c39b031c47ae4f04eab30b4b7"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.331812 9590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:11.343858 9590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:11.344540 9590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:11.344744 9590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:11.345330 9590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:11.346277 9590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:11.346349 9590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.346418 9590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:11.346465 9590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.355199 9590 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:40511
I20260806 16:08:11.355243 9773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:40511 every 8 connection(s)
I20260806 16:08:11.356097 9590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/data/info.pb
I20260806 16:08:11.365502 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 9590
I20260806 16:08:11.365538 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.358146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38323 (local address 127.13.148.62:40999)
0806 16:08:11.358318 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:11.358322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:11.359310 (+ 988us) server_negotiation.cc:407] Connection header received
0806 16:08:11.360772 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:11.360776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:11.360842 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:11.360940 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:11.361826 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.362374 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.363548 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.363772 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.363883 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.365014 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:11.365018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:11.365020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:11.365038 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:11.365057 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:11.365065 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:11.365158 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:11.365353 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 16:08:11.365420 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 16:08:11.365658 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-0/wal/instance
I20260806 16:08:11.365968 13904 external_mini_cluster.cc:1428] Setting key 13d9e8609987d30d633219bbbdd6bd6b
I20260806 16:08:11.366655 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.358445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40999 (local address 127.13.148.1:38323)
0806 16:08:11.359134 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:11.359176 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:11.360549 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:11.361115 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:11.361124 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:11.361206 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:11.361680 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.361692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.362535 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.362537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.363412 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.363422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.363925 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.363930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.364103 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.364755 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:11.364757 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:11.364796 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:11.364858 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:11.365161 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:11.365174 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:11.365182 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:08:11.365323 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 16:08:11.365642 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":130,"threads_started":1}
I20260806 16:08:11.367942 9775 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40999
I20260806 16:08:11.368156 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40999
--builtin_ntp_servers=127.13.148.20:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:08:11.368194 9775 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:11.368729 9775 heartbeater.cc:511] Master 127.13.148.62:40999 requested a full tablet report, sending...
I20260806 16:08:11.370249 9501 ts_manager.cc:194] Registered new tserver with Master: f3ae8791d2a74098b7871c1cab01e34e (127.13.148.1:40511)
I20260806 16:08:11.371460 9501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.1:38323
W20260806 16:08:11.481016 9779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:11.481302 9779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:11.481341 9779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:11.485380 9779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:11.485472 9779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:11.485587 9779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260806 16:08:11.490765 9779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:11.491940 9779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:11.493312 9779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:11.503062 9787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:11.503187 9779 server_base.cc:1034] running on GCE node
W20260806 16:08:11.503039 9785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:11.503034 9784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:11.503769 9779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:11.504441 9779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:11.505642 9779 hybrid_clock.cc:648] HybridClock initialized: now 1786032491505610 us; error 41 us; skew 500 ppm
I20260806 16:08:11.507897 9779 webserver.cc:460] Webserver started at http://127.13.148.2:41803/ using document root <none> and password file <none>
I20260806 16:08:11.508527 9779 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:11.508628 9779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:11.508885 9779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:11.511193 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/data/instance:
uuid: "ef5ec4ad8e194a4ab00b8483df321af5"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "1bbf2f378b0b54934b994f3f8b49dd03"
server_key_iv: "df2198745a7b53f7499bd69dbd63aa8f"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.511811 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "ef5ec4ad8e194a4ab00b8483df321af5"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "1bbf2f378b0b54934b994f3f8b49dd03"
server_key_iv: "df2198745a7b53f7499bd69dbd63aa8f"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.515893 9779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:11.518518 9796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.519896 9779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:08:11.520042 9779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/wal
uuid: "ef5ec4ad8e194a4ab00b8483df321af5"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "1bbf2f378b0b54934b994f3f8b49dd03"
server_key_iv: "df2198745a7b53f7499bd69dbd63aa8f"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.520164 9779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:11.537122 9779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:11.537873 9779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:11.538091 9779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:11.538842 9779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:11.539875 9779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:11.539951 9779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.540030 9779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:11.540081 9779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.549477 9779 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:44485
I20260806 16:08:11.549508 9965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:44485 every 8 connection(s)
I20260806 16:08:11.550422 9779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/data/info.pb
I20260806 16:08:11.556039 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 9779
I20260806 16:08:11.556187 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/wal/instance
I20260806 16:08:11.556504 13904 external_mini_cluster.cc:1428] Setting key 3195051da1217eb961b36515a163f729
I20260806 16:08:11.558657 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40999
--builtin_ntp_servers=127.13.148.20:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 16:08:11.560905 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.552608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:59941 (local address 127.13.148.62:40999)
0806 16:08:11.552804 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:11.552809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:11.553862 (+ 1053us) server_negotiation.cc:407] Connection header received
0806 16:08:11.555355 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:11.555358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:11.555418 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:11.555521 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:11.556727 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.557271 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.558702 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.559073 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.559200 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.560395 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:11.560402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:11.560406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:11.560423 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:11.560450 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:11.560462 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:11.560548 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:11.560736 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 16:08:11.560786 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 16:08:11.562111 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.553049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40999 (local address 127.13.148.2:59941)
0806 16:08:11.553694 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:11.553734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:11.555139 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:11.555689 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:11.555696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:11.555813 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:11.556516 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.556541 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.557401 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.557405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.558578 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.558588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.559216 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.559219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.559388 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.560082 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:11.560087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:11.560143 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:11.560216 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:11.560567 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:11.560578 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:11.560586 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 16:08:11.560726 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 16:08:11.561038 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260806 16:08:11.563265 9966 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40999
I20260806 16:08:11.563513 9966 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:11.564106 9966 heartbeater.cc:511] Master 127.13.148.62:40999 requested a full tablet report, sending...
I20260806 16:08:11.565155 9501 ts_manager.cc:194] Registered new tserver with Master: ef5ec4ad8e194a4ab00b8483df321af5 (127.13.148.2:44485)
I20260806 16:08:11.565738 9501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.2:59941
W20260806 16:08:11.681907 9969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:11.682194 9969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:11.682279 9969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:11.686427 9969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:11.686523 9969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:11.686617 9969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260806 16:08:11.691424 9969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:11.692512 9969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:11.693933 9969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:11.703940 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:11.703965 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:11.704128 9969 server_base.cc:1034] running on GCE node
W20260806 16:08:11.703962 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:11.704704 9969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:11.705372 9969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:11.706580 9969 hybrid_clock.cc:648] HybridClock initialized: now 1786032491706552 us; error 38 us; skew 500 ppm
I20260806 16:08:11.708593 9969 webserver.cc:460] Webserver started at http://127.13.148.3:40295/ using document root <none> and password file <none>
I20260806 16:08:11.709151 9969 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:11.709211 9969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:11.709379 9969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:11.711522 9969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/data/instance:
uuid: "1bd84dcdf41a440ca713c911683940c2"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "2e24b4d58af749f7c0dc4c4dd3108b2b"
server_key_iv: "4ce84ffd0d9c7044b556dd2cd152b04e"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.712041 9969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "1bd84dcdf41a440ca713c911683940c2"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "2e24b4d58af749f7c0dc4c4dd3108b2b"
server_key_iv: "4ce84ffd0d9c7044b556dd2cd152b04e"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.715740 9969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:08:11.718077 9988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.719309 9969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:11.719408 9969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/wal
uuid: "1bd84dcdf41a440ca713c911683940c2"
format_stamp: "Formatted at 2026-08-06 16:08:11 on dist-test-slave-7kzw"
server_key: "2e24b4d58af749f7c0dc4c4dd3108b2b"
server_key_iv: "4ce84ffd0d9c7044b556dd2cd152b04e"
server_key_version: "encryptionkey@0"
I20260806 16:08:11.719492 9969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:11.739981 9969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:11.740633 9969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:11.740793 9969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:11.741304 9969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 16:08:11.742192 9969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 16:08:11.742238 9969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.742278 9969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 16:08:11.742300 9969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:11.750815 9969 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:33909
I20260806 16:08:11.750898 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:33909 every 8 connection(s)
I20260806 16:08:11.751754 9969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/data/info.pb
I20260806 16:08:11.759120 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 9969
I20260806 16:08:11.759269 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/wal/instance
I20260806 16:08:11.759550 13904 external_mini_cluster.cc:1428] Setting key 040e9effa0dd63ddeaf66667f93aa101
I20260806 16:08:11.762109 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.753639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:35161 (local address 127.13.148.62:40999)
0806 16:08:11.753802 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:11.753806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:11.754832 (+ 1026us) server_negotiation.cc:407] Connection header received
0806 16:08:11.756420 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:11.756424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:11.756493 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:11.756611 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:11.757664 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.758248 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.759677 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.759991 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.760122 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.761513 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:11.761518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:11.761521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:11.761534 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:11.761558 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:11.761569 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:11.761656 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:11.761851 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 16:08:11.761919 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 16:08:11.763049 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.753948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40999 (local address 127.13.148.3:35161)
0806 16:08:11.754654 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:11.754705 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:11.756174 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:11.756800 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:11.756807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:11.756898 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:11.757487 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.757502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.758393 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.758402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.759534 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.759545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.760212 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.760217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.760417 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.761226 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 16:08:11.761231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:11.761285 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 16:08:11.761357 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:11.761679 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:11.761696 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:11.761701 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 16:08:11.761818 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 16:08:11.762190 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":129,"threads_started":1}
I20260806 16:08:11.764241 10155 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40999
I20260806 16:08:11.764483 10155 heartbeater.cc:464] Registering TS with master...
I20260806 16:08:11.764981 10155 heartbeater.cc:511] Master 127.13.148.62:40999 requested a full tablet report, sending...
I20260806 16:08:11.766062 9501 ts_manager.cc:194] Registered new tserver with Master: 1bd84dcdf41a440ca713c911683940c2 (127.13.148.3:33909)
I20260806 16:08:11.766825 9501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.3:35161
I20260806 16:08:11.774195 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 16:08:11.780489 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.776284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.13.148.62:40999)
0806 16:08:11.776486 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:11.776491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:11.776649 (+ 158us) server_negotiation.cc:407] Connection header received
0806 16:08:11.776800 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:11.776803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:11.776853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:11.776944 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:11.777251 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.777809 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.778816 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.779051 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.779168 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.780040 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:11.780044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:11.780046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:11.780064 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:11.780081 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:11.780090 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:11.780191 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:11.780293 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 16:08:11.780345 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 16:08:11.783588 9501 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 16:08:11.786731 9501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 16:08:11.801558 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.795519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.13.148.2:44485)
0806 16:08:11.796855 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:11.796870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:11.796896 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:08:11.796977 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:11.796982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:11.797189 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:11.797375 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:11.797748 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.798995 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.800331 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.800870 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.800993 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.801212 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:11.801281 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 16:08:11.801382 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":163,"threads_started":1}
I20260806 16:08:11.801687 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.795369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:44485 (local address 127.0.0.1:44506)
0806 16:08:11.796348 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:11.796403 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:11.796543 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:11.797377 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:11.797385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:11.797425 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:11.797611 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.797619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.799121 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.799124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.800217 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.800225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.800993 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.800997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.801102 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.801138 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 16:08:11.801350 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 16:08:11.801486 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":23,"spinlock_wait_cycles":16000,"thread_start_us":93,"threads_started":1}
I20260806 16:08:11.801759 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.795664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:33909 (local address 127.0.0.1:46942)
0806 16:08:11.796580 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:11.796608 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:11.796698 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:11.797386 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:11.797389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:11.797427 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:11.797620 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.797627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.799189 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.799193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.800260 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.800271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.801291 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.801295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.801387 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.801410 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 16:08:11.801460 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 16:08:11.801532 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":23,"spinlock_wait_cycles":4096,"thread_start_us":90,"threads_started":1}
I20260806 16:08:11.802009 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.796209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.13.148.1:40511)
0806 16:08:11.796971 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:11.796987 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:11.797013 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:08:11.797106 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:11.797111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:11.797268 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:11.797439 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:11.797826 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.798633 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.800182 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.801002 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.801115 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.801328 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:11.801756 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 16:08:11.801862 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":119,"threads_started":1}
I20260806 16:08:11.802277 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.795585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:40511 (local address 127.0.0.1:58992)
0806 16:08:11.796348 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:11.796403 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:11.796543 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:11.797594 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:11.797596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:11.797611 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:11.797710 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.797719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.798739 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.798745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.800039 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:11.800058 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.801381 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.801384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:11.801478 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.801496 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 16:08:11.801558 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 16:08:11.801617 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":3,"spinlock_wait_cycles":5632,"thread_start_us":107,"threads_started":1}
I20260806 16:08:11.802717 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:11.796132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.13.148.3:33909)
0806 16:08:11.796858 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:11.796870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:11.796896 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:08:11.796977 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:11.796982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:11.797234 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:11.797788 (+ 554us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:11.797927 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.799037 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.800404 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:11.801154 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:11.802057 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:11.802379 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:11.802433 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 16:08:11.802553 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":163,"threads_started":1}
I20260806 16:08:11.804904 9680 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa09bfc414b4346ade2efb2df3cffff (DEFAULT_TABLE table=test-table [id=e7876fc37dad4e93bbb749bb4b336ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:11.805034 9872 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa09bfc414b4346ade2efb2df3cffff (DEFAULT_TABLE table=test-table [id=e7876fc37dad4e93bbb749bb4b336ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:11.805346 10054 tablet_service.cc:1467] Processing CreateTablet for tablet 8aa09bfc414b4346ade2efb2df3cffff (DEFAULT_TABLE table=test-table [id=e7876fc37dad4e93bbb749bb4b336ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 16:08:11.805994 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa09bfc414b4346ade2efb2df3cffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:11.806067 9872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa09bfc414b4346ade2efb2df3cffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:11.806221 10054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aa09bfc414b4346ade2efb2df3cffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:11.812500 10177 tablet_bootstrap.cc:492] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e: Bootstrap starting.
I20260806 16:08:11.813652 10179 tablet_bootstrap.cc:492] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Bootstrap starting.
I20260806 16:08:11.814812 10178 tablet_bootstrap.cc:492] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Bootstrap starting.
I20260806 16:08:11.815238 10177 tablet_bootstrap.cc:654] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:11.815646 10179 tablet_bootstrap.cc:654] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:11.816045 10177 log.cc:824] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:11.816455 10179 log.cc:824] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:11.816941 10178 tablet_bootstrap.cc:654] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:11.817800 10178 log.cc:824] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:11.818413 10179 tablet_bootstrap.cc:492] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: No bootstrap required, opened a new log
I20260806 16:08:11.818410 10177 tablet_bootstrap.cc:492] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e: No bootstrap required, opened a new log
I20260806 16:08:11.818665 10177 ts_tablet_manager.cc:1397] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260806 16:08:11.818670 10179 ts_tablet_manager.cc:1397] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260806 16:08:11.819749 10178 tablet_bootstrap.cc:492] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: No bootstrap required, opened a new log
I20260806 16:08:11.819927 10178 ts_tablet_manager.cc:1397] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 16:08:11.822229 10177 raft_consensus.cc:348] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:11.822552 10177 raft_consensus.cc:374] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:11.822664 10177 raft_consensus.cc:729] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ae8791d2a74098b7871c1cab01e34e, State: Initialized, Role: FOLLOWER
I20260806 16:08:11.823124 10177 consensus_queue.cc:260] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:11.823042 10179 raft_consensus.cc:348] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:11.823287 10179 raft_consensus.cc:374] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:11.823433 10179 raft_consensus.cc:729] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd84dcdf41a440ca713c911683940c2, State: Initialized, Role: FOLLOWER
I20260806 16:08:11.823876 9775 heartbeater.cc:503] Master 127.13.148.62:40999 was elected leader, sending a full tablet report...
I20260806 16:08:11.823870 10179 consensus_queue.cc:260] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:11.823975 10178 raft_consensus.cc:348] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:11.824208 10178 raft_consensus.cc:374] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:11.824261 10178 raft_consensus.cc:729] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5ec4ad8e194a4ab00b8483df321af5, State: Initialized, Role: FOLLOWER
I20260806 16:08:11.824688 10178 consensus_queue.cc:260] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:11.825189 10179 ts_tablet_manager.cc:1428] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260806 16:08:11.825351 10177 ts_tablet_manager.cc:1428] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260806 16:08:11.825512 10155 heartbeater.cc:503] Master 127.13.148.62:40999 was elected leader, sending a full tablet report...
I20260806 16:08:11.826138 9966 heartbeater.cc:503] Master 127.13.148.62:40999 was elected leader, sending a full tablet report...
I20260806 16:08:11.826329 10178 ts_tablet_manager.cc:1428] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
W20260806 16:08:11.857313 9776 tablet.cc:2369] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:08:12.002800 10159 tablet.cc:2369] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 16:08:12.051682 9967 tablet.cc:2369] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:08:12.160198 10186 raft_consensus.cc:482] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 16:08:12.160422 10186 raft_consensus.cc:504] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:12.161536 10186 leader_election.cc:288] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3ae8791d2a74098b7871c1cab01e34e (127.13.148.1:40511), 1bd84dcdf41a440ca713c911683940c2 (127.13.148.3:33909)
I20260806 16:08:12.165436 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:12.161950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:37831 (local address 127.13.148.3:33909)
0806 16:08:12.162082 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:12.162087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:12.162304 (+ 217us) server_negotiation.cc:407] Connection header received
0806 16:08:12.162408 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:12.162410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:12.162449 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:12.162567 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:12.162903 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.163380 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.164458 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.164854 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.164980 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:12.165033 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:12.165235 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 16:08:12.165307 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:08:12.165453 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:12.161773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:40511 (local address 127.13.148.2:43143)
0806 16:08:12.162117 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:12.162132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:12.162233 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:12.162544 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:12.162549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:12.162571 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:12.162725 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:12.162731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.163484 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.163490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:12.164339 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:12.164349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.165019 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.165023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:12.165133 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:12.165150 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 16:08:12.165243 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 16:08:12.165307 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":65,"thread_start_us":92,"threads_started":1}
I20260806 16:08:12.165462 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:12.161962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:43143 (local address 127.13.148.1:40511)
0806 16:08:12.162093 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:12.162098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:12.162246 (+ 148us) server_negotiation.cc:407] Connection header received
0806 16:08:12.162344 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:12.162351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:12.162414 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:12.162526 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 16:08:12.162841 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.163369 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.164497 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.164867 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.164997 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:12.165047 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:12.165278 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 16:08:12.165355 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:08:12.165462 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:12.161813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:33909 (local address 127.13.148.2:37831)
0806 16:08:12.162171 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:12.162204 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:12.162322 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:12.162592 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:12.162595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:12.162624 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 16:08:12.162769 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:12.162793 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.163514 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.163517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:12.164332 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:12.164341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.164979 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.164983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:12.165086 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:12.165129 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 16:08:12.165189 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 16:08:12.165276 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":27,"spinlock_wait_cycles":1280,"thread_start_us":71,"threads_started":1}
I20260806 16:08:12.166222 9712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa09bfc414b4346ade2efb2df3cffff" candidate_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ae8791d2a74098b7871c1cab01e34e" is_pre_election: true
I20260806 16:08:12.166404 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa09bfc414b4346ade2efb2df3cffff" candidate_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd84dcdf41a440ca713c911683940c2" is_pre_election: true
I20260806 16:08:12.166618 9712 raft_consensus.cc:2454] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5ec4ad8e194a4ab00b8483df321af5 in term 0.
I20260806 16:08:12.166745 10087 raft_consensus.cc:2454] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5ec4ad8e194a4ab00b8483df321af5 in term 0.
I20260806 16:08:12.167137 9799 leader_election.cc:302] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef5ec4ad8e194a4ab00b8483df321af5, f3ae8791d2a74098b7871c1cab01e34e; no voters:
I20260806 16:08:12.167455 10186 raft_consensus.cc:2781] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 16:08:12.167544 10186 raft_consensus.cc:482] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 16:08:12.167591 10186 raft_consensus.cc:3037] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:12.168612 10186 raft_consensus.cc:504] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:12.168943 10186 leader_election.cc:288] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [CANDIDATE]: Term 1 election: Requested vote from peers f3ae8791d2a74098b7871c1cab01e34e (127.13.148.1:40511), 1bd84dcdf41a440ca713c911683940c2 (127.13.148.3:33909)
I20260806 16:08:12.169323 10087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa09bfc414b4346ade2efb2df3cffff" candidate_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd84dcdf41a440ca713c911683940c2"
I20260806 16:08:12.169344 9712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aa09bfc414b4346ade2efb2df3cffff" candidate_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ae8791d2a74098b7871c1cab01e34e"
I20260806 16:08:12.169451 10087 raft_consensus.cc:3037] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:12.169461 9712 raft_consensus.cc:3037] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:12.170455 9712 raft_consensus.cc:2454] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5ec4ad8e194a4ab00b8483df321af5 in term 1.
I20260806 16:08:12.170533 10087 raft_consensus.cc:2454] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5ec4ad8e194a4ab00b8483df321af5 in term 1.
I20260806 16:08:12.170800 9799 leader_election.cc:302] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef5ec4ad8e194a4ab00b8483df321af5, f3ae8791d2a74098b7871c1cab01e34e; no voters:
I20260806 16:08:12.170969 10186 raft_consensus.cc:2781] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:12.171214 10186 raft_consensus.cc:686] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 1 LEADER]: Becoming Leader. State: Replica: ef5ec4ad8e194a4ab00b8483df321af5, State: Running, Role: LEADER
I20260806 16:08:12.171531 10186 consensus_queue.cc:237] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } }
I20260806 16:08:12.174755 9501 catalog_manager.cc:5495] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5ec4ad8e194a4ab00b8483df321af5 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5ec4ad8e194a4ab00b8483df321af5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 44485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 } health_report { overall_health: UNKNOWN } } }
I20260806 16:08:12.255129 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:12.251134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.13.148.2:44485)
0806 16:08:12.251292 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:12.251297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:12.251314 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:08:12.251591 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:12.251594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:12.251669 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:12.251774 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:12.252023 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.252565 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.253452 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:12.253763 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:12.253887 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:12.254150 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:12.254661 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:12.254767 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:12.254888 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 16:08:12.254938 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 16:08:12.261304 9712 raft_consensus.cc:1261] T 8aa09bfc414b4346ade2efb2df3cffff P f3ae8791d2a74098b7871c1cab01e34e [term 1 FOLLOWER]: Refusing update from remote peer ef5ec4ad8e194a4ab00b8483df321af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:12.261303 10087 raft_consensus.cc:1261] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 1 FOLLOWER]: Refusing update from remote peer ef5ec4ad8e194a4ab00b8483df321af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:12.262035 10186 consensus_queue.cc:1035] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3ae8791d2a74098b7871c1cab01e34e" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:12.262290 10189 consensus_queue.cc:1035] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd84dcdf41a440ca713c911683940c2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:12.280820 9501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:12.281037 9501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 16:08:12.283529 9501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b67dbc124b9b4d588ffc9e011078dd70: Sending DeleteTablet for 3 replicas of tablet 8aa09bfc414b4346ade2efb2df3cffff
I20260806 16:08:12.284402 9680 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aa09bfc414b4346ade2efb2df3cffff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:12 UTC) from {username='slave'} at 127.0.0.1:58992
I20260806 16:08:12.284483 10054 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aa09bfc414b4346ade2efb2df3cffff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:12 UTC) from {username='slave'} at 127.0.0.1:46942
I20260806 16:08:12.284469 9872 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aa09bfc414b4346ade2efb2df3cffff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 16:08:12 UTC) from {username='slave'} at 127.0.0.1:44506
I20260806 16:08:12.284967 10208 tablet_replica.cc:330] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: stopping tablet replica
I20260806 16:08:12.285316 10208 raft_consensus.cc:2229] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 16:08:12.285388 10209 tablet_replica.cc:330] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: stopping tablet replica
I20260806 16:08:12.285702 10208 raft_consensus.cc:2258] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:12.285849 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 9590
I20260806 16:08:12.290238 10209 raft_consensus.cc:2229] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 16:08:12.290961 10209 raft_consensus.cc:2258] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 16:08:12.292445 10208 ts_tablet_manager.cc:1905] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:12.294827 10209 ts_tablet_manager.cc:1905] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 16:08:12.295718 10208 ts_tablet_manager.cc:1918] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:12.295840 10208 log.cc:1196] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-2/wal/wals/8aa09bfc414b4346ade2efb2df3cffff
I20260806 16:08:12.296232 10208 ts_tablet_manager.cc:1939] T 8aa09bfc414b4346ade2efb2df3cffff P 1bd84dcdf41a440ca713c911683940c2: Deleting consensus metadata
I20260806 16:08:12.297712 9478 catalog_manager.cc:4841] TS 1bd84dcdf41a440ca713c911683940c2 (127.13.148.3:33909): tablet 8aa09bfc414b4346ade2efb2df3cffff (table test-table [id=e7876fc37dad4e93bbb749bb4b336ee7]) successfully deleted
I20260806 16:08:12.298384 10209 ts_tablet_manager.cc:1918] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 16:08:12.298540 10209 log.cc:1196] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Deleting WAL directory at /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786032227601881-13904-0/minicluster-data/ts-1/wal/wals/8aa09bfc414b4346ade2efb2df3cffff
I20260806 16:08:12.298883 10209 ts_tablet_manager.cc:1939] T 8aa09bfc414b4346ade2efb2df3cffff P ef5ec4ad8e194a4ab00b8483df321af5: Deleting consensus metadata
I20260806 16:08:12.299952 9482 catalog_manager.cc:4841] TS ef5ec4ad8e194a4ab00b8483df321af5 (127.13.148.2:44485): tablet 8aa09bfc414b4346ade2efb2df3cffff (table test-table [id=e7876fc37dad4e93bbb749bb4b336ee7]) successfully deleted
W20260806 16:08:12.304050 9479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:40511 (error 108)
I20260806 16:08:12.304175 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 9779
I20260806 16:08:12.310220 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:12.308510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:40511 (local address 127.0.0.1:59000)
0806 16:08:12.310053 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:12.310125 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":74,"threads_started":1}
W20260806 16:08:12.310396 9479 catalog_manager.cc:4584] TS f3ae8791d2a74098b7871c1cab01e34e (127.13.148.1:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8aa09bfc414b4346ade2efb2df3cffff: Network error: Client connection negotiation failed: client connection to 127.13.148.1:40511: connect: Connection refused (error 111)
I20260806 16:08:12.315407 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 9969
I20260806 16:08:12.327019 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 9454
2026-08-06T16:08:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1463 ms)
[----------] 6 tests from EncryptionPolicyTest (12198 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:12 dist-test-slave-7kzw krb5kdc[10217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:12 dist-test-slave-7kzw krb5kdc[10217](info): set up 2 sockets
Aug 06 16:08:12 dist-test-slave-7kzw krb5kdc[10217](info): commencing operation
krb5kdc: starting...
W20260806 16:08:13.576570 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:13 dist-test-slave-7kzw krb5kdc[10217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:13.717905 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:13.837191 10233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:13.837524 10233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:13.837611 10233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:13.841616 10233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:13.841717 10233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:13.841760 10233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:13.841797 10233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:13.841832 10233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:13.847127 10233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:13.848263 10233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:13.849300 10233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:13.858199 10239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:13.858358 10233 server_base.cc:1034] running on GCE node
W20260806 16:08:13.858280 10245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:13.858202 10248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:13.859177 10233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:13.860343 10233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:13.861552 10233 hybrid_clock.cc:648] HybridClock initialized: now 1786032493861509 us; error 60 us; skew 500 ppm
Aug 06 16:08:13 dist-test-slave-7kzw krb5kdc[10217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:13.864640 10233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:13.865628 10233 webserver.cc:460] Webserver started at http://127.0.0.1:46209/ using document root <none> and password file <none>
I20260806 16:08:13.866211 10233 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:13.866307 10233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:13.866614 10233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:13.868775 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "b9ea7842ed8241ea876208fa3d841c53"
format_stamp: "Formatted at 2026-08-06 16:08:13 on dist-test-slave-7kzw"
server_key: "8c00f1b8523e622619a34e4ae9027115"
server_key_iv: "33ecba3d9203380807c83ed61e5c4c32"
server_key_version: "encryptionkey@0"
I20260806 16:08:13.869407 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "b9ea7842ed8241ea876208fa3d841c53"
format_stamp: "Formatted at 2026-08-06 16:08:13 on dist-test-slave-7kzw"
server_key: "8c00f1b8523e622619a34e4ae9027115"
server_key_iv: "33ecba3d9203380807c83ed61e5c4c32"
server_key_version: "encryptionkey@0"
I20260806 16:08:13.873317 10233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 16:08:13.876013 10260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:13.877550 10233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:08:13.877841 10233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "b9ea7842ed8241ea876208fa3d841c53"
format_stamp: "Formatted at 2026-08-06 16:08:13 on dist-test-slave-7kzw"
server_key: "8c00f1b8523e622619a34e4ae9027115"
server_key_iv: "33ecba3d9203380807c83ed61e5c4c32"
server_key_version: "encryptionkey@0"
I20260806 16:08:13.877986 10233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:13.910086 10233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:13.913666 10233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:13.913899 10233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:13.921588 10233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32953
I20260806 16:08:13.921619 10335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32953 every 8 connection(s)
I20260806 16:08:13.922667 10233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:13.925546 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:13.926259 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 10233
I20260806 16:08:13.926415 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:13.926789 13904 external_mini_cluster.cc:1428] Setting key a62adb927814480c33896460c3285b3f
I20260806 16:08:13.932099 10336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53: Bootstrap starting.
I20260806 16:08:13.934587 10336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:13.935449 10336 log.cc:824] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:08:13 dist-test-slave-7kzw krb5kdc[10217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 16:08:13.937937 10336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53: No bootstrap required, opened a new log
I20260806 16:08:13.942970 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:13.928375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48881 (local address 127.0.0.1:32953)
0806 16:08:13.928881 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:13.928897 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:13.928934 (+ 37us) server_negotiation.cc:407] Connection header received
0806 16:08:13.929611 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:13.929636 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:13.929982 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:13.930331 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:13.931232 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:13.932215 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:13.933118 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:13.933599 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:13.933779 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:13.937288 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:13.937294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:13.937311 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:13.937345 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:13.939452 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:13.940025 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:13.940032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:13.940040 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:13.940210 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:13.940559 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:13.940563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:13.940568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:13.941111 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:13.941335 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:13.941662 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 16:08:13.941907 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":170,"threads_started":1}
I20260806 16:08:13.942905 10336 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea7842ed8241ea876208fa3d841c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } }
I20260806 16:08:13.943198 10336 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:13.943274 10336 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ea7842ed8241ea876208fa3d841c53, State: Initialized, Role: FOLLOWER
I20260806 16:08:13.943759 10336 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea7842ed8241ea876208fa3d841c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } }
I20260806 16:08:13.943924 10336 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:13.944015 10336 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:13.944932 10336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:13.946110 10336 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea7842ed8241ea876208fa3d841c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } }
I20260806 16:08:13.946199 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:13.946508 10336 leader_election.cc:302] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ea7842ed8241ea876208fa3d841c53; no voters:
I20260806 16:08:13.946872 10336 leader_election.cc:288] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:13.946952 10344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:13.947157 10344 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [term 1 LEADER]: Becoming Leader. State: Replica: b9ea7842ed8241ea876208fa3d841c53, State: Running, Role: LEADER
I20260806 16:08:13.947474 10344 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea7842ed8241ea876208fa3d841c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } }
I20260806 16:08:13.948232 10336 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:13.949115 10345 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ea7842ed8241ea876208fa3d841c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea7842ed8241ea876208fa3d841c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } } }
I20260806 16:08:13.949268 10345 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:13.949398 10346 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ea7842ed8241ea876208fa3d841c53. Latest consensus state: current_term: 1 leader_uuid: "b9ea7842ed8241ea876208fa3d841c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea7842ed8241ea876208fa3d841c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32953 } } }
I20260806 16:08:13.949514 10346 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53 [sys.catalog]: This master's current role is: LEADER
W20260806 16:08:13.953428 10371 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:13.953511 10371 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:13.953594 10362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:13.954435 10362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:13.957481 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:13.948664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48679 (local address 127.0.0.1:32953)
0806 16:08:13.948780 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:13.948784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:13.949040 (+ 256us) server_negotiation.cc:407] Connection header received
0806 16:08:13.949196 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:13.949201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:13.949254 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:13.949336 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:13.950373 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:13.951205 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:13.952027 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:13.952271 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:13.952386 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:13.954110 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:13.954119 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:13.954121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:13.954149 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:13.955673 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:13.956182 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:13.956185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:13.956186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:13.956261 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:13.956608 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:13.956611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:13.956613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:13.956787 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:13.956945 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:13.957238 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 16:08:13.957331 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:08:13.960358 10362 catalog_manager.cc:1347] Generated new cluster ID: b19ea6177ca04ca5a891155775e7cd00
I20260806 16:08:13.960424 10362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:13.973893 10362 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:13.975214 10362 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:13.987507 10362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ea7842ed8241ea876208fa3d841c53: Generated new TSK 0
I20260806 16:08:13.988384 10362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:13.997601 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 10233
W20260806 16:08:14.011411 10213 connection.cc:535] client connection to 127.0.0.1:32953 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32953): Connection reset by peer (error 104)
2026-08-06T16:08:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1671 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:14 dist-test-slave-7kzw krb5kdc[10379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:14 dist-test-slave-7kzw krb5kdc[10379](info): set up 2 sockets
Aug 06 16:08:14 dist-test-slave-7kzw krb5kdc[10379](info): commencing operation
krb5kdc: starting...
W20260806 16:08:15.283427 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:15 dist-test-slave-7kzw krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:15.427660 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:15.544024 10397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:15.544348 10397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:15.544435 10397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:15.548595 10397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:15.548691 10397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:15.548753 10397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:15.548808 10397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:15.548847 10397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:15.554109 10397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:15.555295 10397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:15.556365 10397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:15.565132 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:15.565155 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:15.565148 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:15.565761 10397 server_base.cc:1034] running on GCE node
I20260806 16:08:15.566332 10397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:15.567476 10397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:15.568637 10397 hybrid_clock.cc:648] HybridClock initialized: now 1786032495568608 us; error 44 us; skew 500 ppm
Aug 06 16:08:15 dist-test-slave-7kzw krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:15.571540 10397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:15.572443 10397 webserver.cc:460] Webserver started at http://127.0.0.1:34433/ using document root <none> and password file <none>
I20260806 16:08:15.572999 10397 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:15.573060 10397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:15.573230 10397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:15.575304 10397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "83ff5496068548f7943a5c7ebef10819"
format_stamp: "Formatted at 2026-08-06 16:08:15 on dist-test-slave-7kzw"
server_key: "cdc0b504f2b87e0350fd4777c8cd962c"
server_key_iv: "f00930e72e662bc1a466c83f83971cf3"
server_key_version: "encryptionkey@0"
I20260806 16:08:15.575830 10397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "83ff5496068548f7943a5c7ebef10819"
format_stamp: "Formatted at 2026-08-06 16:08:15 on dist-test-slave-7kzw"
server_key: "cdc0b504f2b87e0350fd4777c8cd962c"
server_key_iv: "f00930e72e662bc1a466c83f83971cf3"
server_key_version: "encryptionkey@0"
I20260806 16:08:15.579499 10397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:08:15.581820 10418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:15.583069 10397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:15.583176 10397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "83ff5496068548f7943a5c7ebef10819"
format_stamp: "Formatted at 2026-08-06 16:08:15 on dist-test-slave-7kzw"
server_key: "cdc0b504f2b87e0350fd4777c8cd962c"
server_key_iv: "f00930e72e662bc1a466c83f83971cf3"
server_key_version: "encryptionkey@0"
I20260806 16:08:15.583258 10397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:15.597848 10397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:15.601205 10397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:15.601365 10397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:15.608991 10397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44283
I20260806 16:08:15.609005 10492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44283 every 8 connection(s)
I20260806 16:08:15.610008 10397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:15.612900 10494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:15.615449 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 10397
I20260806 16:08:15.615599 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:15.615919 13904 external_mini_cluster.cc:1428] Setting key e7ea9f2ed89254297ad76d5de2e7bc06
I20260806 16:08:15.619254 10494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819: Bootstrap starting.
I20260806 16:08:15.621502 10494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:15.622269 10494 log.cc:824] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:15.624253 10494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819: No bootstrap required, opened a new log
I20260806 16:08:15.625669 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:15.617480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49859 (local address 127.0.0.1:44283)
0806 16:08:15.618158 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:15.618172 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:15.618210 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:08:15.618912 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:15.618933 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:15.619277 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:15.619708 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:15.620122 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:15.621104 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:15.622077 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:15.622668 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:15.622839 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:15.623806 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:15.623811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:15.623825 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:15.624228 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:15.624285 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:15.624309 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:15.624464 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:15.624778 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 16:08:15.624937 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":139,"threads_started":1}
I20260806 16:08:15.628579 10494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff5496068548f7943a5c7ebef10819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260806 16:08:15.628777 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:15.628868 10494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:15.628953 10494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ff5496068548f7943a5c7ebef10819, State: Initialized, Role: FOLLOWER
I20260806 16:08:15.629420 10494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff5496068548f7943a5c7ebef10819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260806 16:08:15.629578 10494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:15.629668 10494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:15.629767 10494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:15.630889 10494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff5496068548f7943a5c7ebef10819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260806 16:08:15.631285 10494 leader_election.cc:302] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83ff5496068548f7943a5c7ebef10819; no voters:
I20260806 16:08:15.631665 10494 leader_election.cc:288] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:15.631736 10518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:15.632010 10518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [term 1 LEADER]: Becoming Leader. State: Replica: 83ff5496068548f7943a5c7ebef10819, State: Running, Role: LEADER
I20260806 16:08:15.632329 10518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff5496068548f7943a5c7ebef10819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260806 16:08:15.633085 10494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:15.633539 10518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83ff5496068548f7943a5c7ebef10819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff5496068548f7943a5c7ebef10819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } } }
I20260806 16:08:15.633713 10518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:15.633965 10518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83ff5496068548f7943a5c7ebef10819. Latest consensus state: current_term: 1 leader_uuid: "83ff5496068548f7943a5c7ebef10819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ff5496068548f7943a5c7ebef10819" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } } }
I20260806 16:08:15.634137 10518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:15.634459 10523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:15.635795 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:15.631156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41947 (local address 127.0.0.1:44283)
0806 16:08:15.631288 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:15.631292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:15.631577 (+ 285us) server_negotiation.cc:407] Connection header received
0806 16:08:15.631612 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:15.631614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:15.631652 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:15.631737 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:15.632023 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:15.632640 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:15.633611 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:15.633945 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:15.634028 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:15.635227 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:15.635230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:15.635232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:15.635244 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:15.635291 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:15.635305 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:15.635394 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:15.635617 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 16:08:15.635655 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 16:08:15.638615 10523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:15.644407 10523 catalog_manager.cc:1347] Generated new cluster ID: 71583de26b9d4d8e9c5983e221641793
I20260806 16:08:15.644483 10523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:15.663803 10523 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:15.665144 10523 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:15.675145 10523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83ff5496068548f7943a5c7ebef10819: Generated new TSK 0
I20260806 16:08:15.676040 10523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:15.685827 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 10397
W20260806 16:08:15.699348 10377 connection.cc:535] client connection to 127.0.0.1:44283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44283): Connection reset by peer (error 104)
2026-08-06T16:08:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1688 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:15 dist-test-slave-7kzw krb5kdc[10543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:15 dist-test-slave-7kzw krb5kdc[10543](info): set up 2 sockets
Aug 06 16:08:15 dist-test-slave-7kzw krb5kdc[10543](info): commencing operation
krb5kdc: starting...
W20260806 16:08:16.905897 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:16 dist-test-slave-7kzw krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:17.046794 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:17.162822 10563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:17.163106 10563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:17.163149 10563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:17.167130 10563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:17.167198 10563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:17.167218 10563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:17.167234 10563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:17.167250 10563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:17.172473 10563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:17.173453 10563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:17.174592 10563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:17.183331 10572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:17.183342 10570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:17.183321 10569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:17.183678 10563 server_base.cc:1034] running on GCE node
I20260806 16:08:17.184293 10563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:17.185374 10563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:17.186558 10563 hybrid_clock.cc:648] HybridClock initialized: now 1786032497186529 us; error 48 us; skew 500 ppm
Aug 06 16:08:17 dist-test-slave-7kzw krb5kdc[10543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:17.194766 10563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:17.195705 10563 webserver.cc:460] Webserver started at http://127.0.0.1:41449/ using document root <none> and password file <none>
I20260806 16:08:17.196303 10563 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:17.196403 10563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:17.196643 10563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:17.198774 10563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "5879406d6b45445d94da7e5f37d3e2ce"
format_stamp: "Formatted at 2026-08-06 16:08:17 on dist-test-slave-7kzw"
server_key: "59c1105fdae5ee9d548a2c838ce70ecc"
server_key_iv: "25b97fb003337cc96c7f1de90fe2af3f"
server_key_version: "encryptionkey@0"
I20260806 16:08:17.199364 10563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "5879406d6b45445d94da7e5f37d3e2ce"
format_stamp: "Formatted at 2026-08-06 16:08:17 on dist-test-slave-7kzw"
server_key: "59c1105fdae5ee9d548a2c838ce70ecc"
server_key_iv: "25b97fb003337cc96c7f1de90fe2af3f"
server_key_version: "encryptionkey@0"
I20260806 16:08:17.203248 10563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:08:17.205828 10579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:17.207242 10563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:08:17.207425 10563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "5879406d6b45445d94da7e5f37d3e2ce"
format_stamp: "Formatted at 2026-08-06 16:08:17 on dist-test-slave-7kzw"
server_key: "59c1105fdae5ee9d548a2c838ce70ecc"
server_key_iv: "25b97fb003337cc96c7f1de90fe2af3f"
server_key_version: "encryptionkey@0"
I20260806 16:08:17.207549 10563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:17.212738 10563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:17.216344 10563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:17.216570 10563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:17.224355 10563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42375
I20260806 16:08:17.224416 10647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42375 every 8 connection(s)
I20260806 16:08:17.225391 10563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:17.228341 10648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:17.233882 10648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce: Bootstrap starting.
I20260806 16:08:17.234773 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 10563
I20260806 16:08:17.234925 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:17.235277 13904 external_mini_cluster.cc:1428] Setting key 73eb3a75f0cfc4b77ea006a9a6cd24e6
I20260806 16:08:17.236977 10648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:17.237727 10648 log.cc:824] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:17.240121 10648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce: No bootstrap required, opened a new log
I20260806 16:08:17.240754 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:17.236508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37865 (local address 127.0.0.1:42375)
0806 16:08:17.236981 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:17.236995 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:17.237077 (+ 82us) server_negotiation.cc:407] Connection header received
0806 16:08:17.237790 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:17.237843 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:17.238228 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:17.238658 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:17.239472 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:17.239476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:17.239487 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:17.239497 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:17.239547 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:17.239565 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:17.239652 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:17.239796 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 16:08:17.239971 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
I20260806 16:08:17.243856 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:17.244249 10648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5879406d6b45445d94da7e5f37d3e2ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } }
I20260806 16:08:17.244539 10648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:17.244606 10648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5879406d6b45445d94da7e5f37d3e2ce, State: Initialized, Role: FOLLOWER
I20260806 16:08:17.245083 10648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5879406d6b45445d94da7e5f37d3e2ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } }
I20260806 16:08:17.245271 10648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:17.245358 10648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:17.245478 10648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:17.246696 10648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5879406d6b45445d94da7e5f37d3e2ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } }
I20260806 16:08:17.247114 10648 leader_election.cc:302] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5879406d6b45445d94da7e5f37d3e2ce; no voters:
I20260806 16:08:17.247453 10648 leader_election.cc:288] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:17.247581 10661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:17.247871 10661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [term 1 LEADER]: Becoming Leader. State: Replica: 5879406d6b45445d94da7e5f37d3e2ce, State: Running, Role: LEADER
I20260806 16:08:17.248167 10651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:17.246216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40877 (local address 127.0.0.1:42375)
0806 16:08:17.246344 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:17.246348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:17.246588 (+ 240us) server_negotiation.cc:407] Connection header received
0806 16:08:17.246709 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:17.246711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:17.246741 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:17.246811 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:17.247640 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:17.247644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:17.247646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:17.247668 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:17.247703 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:17.247718 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:17.247811 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:17.247987 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 16:08:17.248030 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:08:17.248252 10661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5879406d6b45445d94da7e5f37d3e2ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } }
I20260806 16:08:17.248852 10648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:17.250032 10663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5879406d6b45445d94da7e5f37d3e2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5879406d6b45445d94da7e5f37d3e2ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } } }
I20260806 16:08:17.250168 10663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:17.250631 10676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:17.250684 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5879406d6b45445d94da7e5f37d3e2ce. Latest consensus state: current_term: 1 leader_uuid: "5879406d6b45445d94da7e5f37d3e2ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5879406d6b45445d94da7e5f37d3e2ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42375 } } }
I20260806 16:08:17.251377 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:17.253762 10676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:17.258994 10676 catalog_manager.cc:1347] Generated new cluster ID: eebb52bae6b443d983383825258a0b01
I20260806 16:08:17.259063 10676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:17.267903 10676 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:17.268231 10676 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:17.276016 10676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5879406d6b45445d94da7e5f37d3e2ce: Generated new TSK 0
I20260806 16:08:17.276894 10676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:17.281284 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 10563
2026-08-06T16:08:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1596 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:17 dist-test-slave-7kzw krb5kdc[10691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:17 dist-test-slave-7kzw krb5kdc[10691](info): set up 2 sockets
Aug 06 16:08:17 dist-test-slave-7kzw krb5kdc[10691](info): commencing operation
krb5kdc: starting...
W20260806 16:08:18.497290 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:18 dist-test-slave-7kzw krb5kdc[10691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:18.633002 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:18.747489 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:18.747828 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:18.747905 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:18.751984 10710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:18.752076 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:18.752128 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:18.752187 10710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:18.752226 10710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:18.757846 10710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:18.759055 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:18.760118 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:18.768790 10720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:18.768854 10718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:18.768800 10722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:18.769162 10710 server_base.cc:1034] running on GCE node
I20260806 16:08:18.769790 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:18.770933 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:18.772101 10710 hybrid_clock.cc:648] HybridClock initialized: now 1786032498772075 us; error 40 us; skew 500 ppm
Aug 06 16:08:18 dist-test-slave-7kzw krb5kdc[10691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:18.775192 10710 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:18.776324 10710 webserver.cc:460] Webserver started at http://127.0.0.1:41235/ using document root <none> and password file <none>
I20260806 16:08:18.776968 10710 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:18.777047 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:18.777346 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:18.779754 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "d25fc2ec01374e0c8a501bf10d85bddd"
format_stamp: "Formatted at 2026-08-06 16:08:18 on dist-test-slave-7kzw"
server_key: "f0ccb05adfce234e09f11fc2fb06b977"
server_key_iv: "bc4c3f3b8c27b322153c1b0485c70e49"
server_key_version: "encryptionkey@0"
I20260806 16:08:18.780421 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "d25fc2ec01374e0c8a501bf10d85bddd"
format_stamp: "Formatted at 2026-08-06 16:08:18 on dist-test-slave-7kzw"
server_key: "f0ccb05adfce234e09f11fc2fb06b977"
server_key_iv: "bc4c3f3b8c27b322153c1b0485c70e49"
server_key_version: "encryptionkey@0"
I20260806 16:08:18.784807 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:08:18.787660 10733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:18.788972 10710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 16:08:18.789156 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "d25fc2ec01374e0c8a501bf10d85bddd"
format_stamp: "Formatted at 2026-08-06 16:08:18 on dist-test-slave-7kzw"
server_key: "f0ccb05adfce234e09f11fc2fb06b977"
server_key_iv: "bc4c3f3b8c27b322153c1b0485c70e49"
server_key_version: "encryptionkey@0"
I20260806 16:08:18.789290 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:18.807036 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:18.810684 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:18.810909 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:18.818739 10710 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39681
I20260806 16:08:18.818769 10797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39681 every 8 connection(s)
I20260806 16:08:18.819778 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:18.820473 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 10710
I20260806 16:08:18.820631 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:18.820966 13904 external_mini_cluster.cc:1428] Setting key dae69a70f5e4096423db35e8d12c935d
I20260806 16:08:18.823226 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:18.830734 10798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd: Bootstrap starting.
Aug 06 16:08:18 dist-test-slave-7kzw krb5kdc[10691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 16:08:18.833361 10798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:18.834147 10798 log.cc:824] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:18.836442 10798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd: No bootstrap required, opened a new log
I20260806 16:08:18.836681 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:18.822525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53345 (local address 127.0.0.1:39681)
0806 16:08:18.823051 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:18.823068 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:18.823114 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:08:18.823873 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:18.823891 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:18.824252 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:18.824566 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:18.825508 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:18.826597 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:18.827852 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:18.828346 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:18.828476 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:18.831667 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:18.831672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:18.831685 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:18.831715 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:18.833554 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:18.834015 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:18.834022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:18.834033 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:18.834156 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:18.834448 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:18.834452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:18.834454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:18.835045 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:18.835261 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:18.835591 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 16:08:18.835896 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":143,"threads_started":1}
I20260806 16:08:18.839998 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:18.840854 10798 raft_consensus.cc:348] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25fc2ec01374e0c8a501bf10d85bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } }
I20260806 16:08:18.841111 10798 raft_consensus.cc:374] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:18.841204 10798 raft_consensus.cc:729] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25fc2ec01374e0c8a501bf10d85bddd, State: Initialized, Role: FOLLOWER
I20260806 16:08:18.841663 10798 consensus_queue.cc:260] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25fc2ec01374e0c8a501bf10d85bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } }
I20260806 16:08:18.841790 10798 raft_consensus.cc:388] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:18.841873 10798 raft_consensus.cc:482] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:18.841995 10798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:18.843106 10798 raft_consensus.cc:504] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25fc2ec01374e0c8a501bf10d85bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } }
I20260806 16:08:18.843456 10798 leader_election.cc:302] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d25fc2ec01374e0c8a501bf10d85bddd; no voters:
I20260806 16:08:18.843793 10798 leader_election.cc:288] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:18.843871 10814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:18.844060 10814 raft_consensus.cc:686] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [term 1 LEADER]: Becoming Leader. State: Replica: d25fc2ec01374e0c8a501bf10d85bddd, State: Running, Role: LEADER
I20260806 16:08:18.844439 10814 consensus_queue.cc:237] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25fc2ec01374e0c8a501bf10d85bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } }
I20260806 16:08:18.845149 10798 sys_catalog.cc:564] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:18.846503 10816 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d25fc2ec01374e0c8a501bf10d85bddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25fc2ec01374e0c8a501bf10d85bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } } }
I20260806 16:08:18.846655 10816 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:18.847153 10815 sys_catalog.cc:455] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d25fc2ec01374e0c8a501bf10d85bddd. Latest consensus state: current_term: 1 leader_uuid: "d25fc2ec01374e0c8a501bf10d85bddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d25fc2ec01374e0c8a501bf10d85bddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39681 } } }
I20260806 16:08:18.847353 10815 sys_catalog.cc:458] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:18.848106 10826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:18.851250 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:18.842205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41237 (local address 127.0.0.1:39681)
0806 16:08:18.842349 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:18.842354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:18.842526 (+ 172us) server_negotiation.cc:407] Connection header received
0806 16:08:18.842699 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:18.842702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:18.842754 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:18.842837 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:18.843560 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:18.844389 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:18.845251 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:18.845547 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:18.845650 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:18.847523 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:18.847527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:18.847530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:18.847556 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:18.849403 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:18.849908 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:18.849911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:18.849914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:18.850008 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:18.850316 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:18.850319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:18.850321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:18.850567 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:18.850701 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:18.850972 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 16:08:18.851094 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 16:08:18.851773 10826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:18.857985 10826 catalog_manager.cc:1347] Generated new cluster ID: 4195760e18c54201adef17ae8921b476
I20260806 16:08:18.858088 10826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:18.871567 10826 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:18.872552 10826 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:18.880106 10826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d25fc2ec01374e0c8a501bf10d85bddd: Generated new TSK 0
I20260806 16:08:18.880863 10826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:18.889297 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 10710
2026-08-06T16:08:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1606 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:18 dist-test-slave-7kzw krb5kdc[10838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:18 dist-test-slave-7kzw krb5kdc[10838](info): set up 2 sockets
Aug 06 16:08:18 dist-test-slave-7kzw krb5kdc[10838](info): commencing operation
krb5kdc: starting...
W20260806 16:08:20.128000 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:20 dist-test-slave-7kzw krb5kdc[10838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:20.274379 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:20.386183 10860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:20.386536 10860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:20.386621 10860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:20.390626 10860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:20.390733 10860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:20.390774 10860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:20.390823 10860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:20.390861 10860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:20.396071 10860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:20.397125 10860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:20.398169 10860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:20.406530 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:20.406550 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:20.406838 10860 server_base.cc:1034] running on GCE node
W20260806 16:08:20.406548 10872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:20.407457 10860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:20.408530 10860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:20.409700 10860 hybrid_clock.cc:648] HybridClock initialized: now 1786032500409685 us; error 27 us; skew 500 ppm
Aug 06 16:08:20 dist-test-slave-7kzw krb5kdc[10838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:20.412604 10860 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:20.413522 10860 webserver.cc:460] Webserver started at http://127.0.0.1:35701/ using document root <none> and password file <none>
I20260806 16:08:20.414095 10860 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:20.414186 10860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:20.414417 10860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:20.416455 10860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "ff65fc8f51874772b9dd73ab5ce240c2"
format_stamp: "Formatted at 2026-08-06 16:08:20 on dist-test-slave-7kzw"
server_key: "b8482f54a057a311297028649ecac2e3"
server_key_iv: "cfd4b06225f1577d664a1bd88da97641"
server_key_version: "encryptionkey@0"
I20260806 16:08:20.417030 10860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "ff65fc8f51874772b9dd73ab5ce240c2"
format_stamp: "Formatted at 2026-08-06 16:08:20 on dist-test-slave-7kzw"
server_key: "b8482f54a057a311297028649ecac2e3"
server_key_iv: "cfd4b06225f1577d664a1bd88da97641"
server_key_version: "encryptionkey@0"
I20260806 16:08:20.420746 10860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 16:08:20.423170 10881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:20.424340 10860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:20.424476 10860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "ff65fc8f51874772b9dd73ab5ce240c2"
format_stamp: "Formatted at 2026-08-06 16:08:20 on dist-test-slave-7kzw"
server_key: "b8482f54a057a311297028649ecac2e3"
server_key_iv: "cfd4b06225f1577d664a1bd88da97641"
server_key_version: "encryptionkey@0"
I20260806 16:08:20.424584 10860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:20.436178 10860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:20.439292 10860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:20.439500 10860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:20.446822 10860 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38845
I20260806 16:08:20.446884 10955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38845 every 8 connection(s)
I20260806 16:08:20.447813 10860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:20.450707 10956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:20.451930 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 10860
I20260806 16:08:20.452083 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:20.452391 13904 external_mini_cluster.cc:1428] Setting key 9262057e8a7d893b035a024eb4e0e8c9
I20260806 16:08:20.457147 10956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2: Bootstrap starting.
I20260806 16:08:20.459921 10956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:20.460793 10956 log.cc:824] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:20.461575 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:20.453754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54309 (local address 127.0.0.1:38845)
0806 16:08:20.454149 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:20.454164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:20.454204 (+ 40us) server_negotiation.cc:407] Connection header received
0806 16:08:20.454889 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:20.454908 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:20.455245 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:20.455509 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:20.455828 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:20.456772 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:20.457974 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:20.458452 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:20.458682 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:20.459477 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:20.459482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:20.459496 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:20.460023 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:20.460099 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:20.460125 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:20.460204 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:20.460372 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 16:08:20.460600 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":129,"threads_started":1}
I20260806 16:08:20.462956 10956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2: No bootstrap required, opened a new log
I20260806 16:08:20.464478 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:20.467594 10956 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65fc8f51874772b9dd73ab5ce240c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38845 } }
I20260806 16:08:20.467873 10956 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:20.467950 10956 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff65fc8f51874772b9dd73ab5ce240c2, State: Initialized, Role: FOLLOWER
I20260806 16:08:20.468389 10956 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65fc8f51874772b9dd73ab5ce240c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38845 } }
I20260806 16:08:20.468515 10956 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:20.468604 10956 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:20.468729 10956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:20.469998 10956 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65fc8f51874772b9dd73ab5ce240c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38845 } }
I20260806 16:08:20.470407 10956 leader_election.cc:302] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff65fc8f51874772b9dd73ab5ce240c2; no voters:
I20260806 16:08:20.470826 10956 leader_election.cc:288] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:20.470963 10968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:20.471287 10968 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [term 1 LEADER]: Becoming Leader. State: Replica: ff65fc8f51874772b9dd73ab5ce240c2, State: Running, Role: LEADER
I20260806 16:08:20.471643 10968 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65fc8f51874772b9dd73ab5ce240c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38845 } }
I20260806 16:08:20.471793 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:20.466866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48611 (local address 127.0.0.1:38845)
0806 16:08:20.466983 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:20.466987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:20.467151 (+ 164us) server_negotiation.cc:407] Connection header received
0806 16:08:20.467280 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:20.467282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:20.467319 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:20.467383 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:20.467877 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:20.468605 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:20.469775 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:20.470012 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:20.470099 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:20.471099 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:20.471103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:20.471106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:20.471122 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:20.471152 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:20.471167 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:20.471245 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:20.471589 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 16:08:20.471661 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:08:20.472314 10956 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:20.473192 10968 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff65fc8f51874772b9dd73ab5ce240c2. Latest consensus state: current_term: 1 leader_uuid: "ff65fc8f51874772b9dd73ab5ce240c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65fc8f51874772b9dd73ab5ce240c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38845 } } }
I20260806 16:08:20.473379 10968 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:20.473690 10970 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff65fc8f51874772b9dd73ab5ce240c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff65fc8f51874772b9dd73ab5ce240c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38845 } } }
I20260806 16:08:20.473820 10981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:20.473855 10970 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:20.477193 10981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:20.482579 10981 catalog_manager.cc:1347] Generated new cluster ID: 4ab75eba73774175a39eaad8825ca877
I20260806 16:08:20.482652 10981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:20.498777 10981 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:20.499764 10981 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:20.504778 10981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff65fc8f51874772b9dd73ab5ce240c2: Generated new TSK 0
I20260806 16:08:20.505592 10981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:20.519140 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 10860
2026-08-06T16:08:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1627 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:20 dist-test-slave-7kzw krb5kdc[10997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:20 dist-test-slave-7kzw krb5kdc[10997](info): set up 2 sockets
Aug 06 16:08:20 dist-test-slave-7kzw krb5kdc[10997](info): commencing operation
krb5kdc: starting...
W20260806 16:08:21.822376 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:21 dist-test-slave-7kzw krb5kdc[10997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:21.957372 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:22.071285 11015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:22.071573 11015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:22.071650 11015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:22.075659 11015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:22.075743 11015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:22.075788 11015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:22.075846 11015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:22.075887 11015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:22.081063 11015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:22.082111 11015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:22.083217 11015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:22.092027 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:22.092061 11022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:22.092238 11015 server_base.cc:1034] running on GCE node
W20260806 16:08:22.092063 11024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:22.092880 11015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:22.093956 11015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:22.095189 11015 hybrid_clock.cc:648] HybridClock initialized: now 1786032502095174 us; error 24 us; skew 500 ppm
Aug 06 16:08:22 dist-test-slave-7kzw krb5kdc[10997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:22.098065 11015 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:22.098990 11015 webserver.cc:460] Webserver started at http://127.0.0.1:44043/ using document root <none> and password file <none>
I20260806 16:08:22.099548 11015 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:22.099653 11015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:22.099884 11015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:22.101892 11015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25"
format_stamp: "Formatted at 2026-08-06 16:08:22 on dist-test-slave-7kzw"
server_key: "bb775697c734bc45771e02791aa8e3da"
server_key_iv: "a2e601fcc4d78945473a1c0a2ad75cf6"
server_key_version: "encryptionkey@0"
I20260806 16:08:22.102499 11015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25"
format_stamp: "Formatted at 2026-08-06 16:08:22 on dist-test-slave-7kzw"
server_key: "bb775697c734bc45771e02791aa8e3da"
server_key_iv: "a2e601fcc4d78945473a1c0a2ad75cf6"
server_key_version: "encryptionkey@0"
I20260806 16:08:22.106158 11015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:08:22.108873 11037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:22.110143 11015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 16:08:22.110306 11015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25"
format_stamp: "Formatted at 2026-08-06 16:08:22 on dist-test-slave-7kzw"
server_key: "bb775697c734bc45771e02791aa8e3da"
server_key_iv: "a2e601fcc4d78945473a1c0a2ad75cf6"
server_key_version: "encryptionkey@0"
I20260806 16:08:22.110430 11015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:22.115106 11015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:22.118110 11015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:22.118314 11015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:22.125483 11015 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36217
I20260806 16:08:22.125494 11110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36217 every 8 connection(s)
I20260806 16:08:22.126415 11015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:22.129261 11111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:22.134338 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 11015
I20260806 16:08:22.134541 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:22.134837 13904 external_mini_cluster.cc:1428] Setting key 915d7cbded1e966f5d3428533082c9f0
I20260806 16:08:22.134902 11111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25: Bootstrap starting.
I20260806 16:08:22.137395 11111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:22.138131 11111 log.cc:824] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:22.139932 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:22.136230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40813 (local address 127.0.0.1:36217)
0806 16:08:22.136687 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:22.136702 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:22.136737 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:08:22.137350 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:22.137367 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:22.137689 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:22.138018 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:22.138760 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:22.138765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:22.138776 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:22.138786 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:22.138833 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:22.138852 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:22.138969 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:22.139077 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 16:08:22.139233 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
I20260806 16:08:22.140340 11111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25: No bootstrap required, opened a new log
I20260806 16:08:22.142632 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:22.144443 11111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } }
I20260806 16:08:22.144740 11111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:22.144831 11111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27060d5e36a14d9bbf11e8d8ba0a8d25, State: Initialized, Role: FOLLOWER
I20260806 16:08:22.145292 11111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } }
I20260806 16:08:22.145473 11111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:22.145577 11111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:22.145714 11111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:22.146684 11111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } }
I20260806 16:08:22.146833 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:22.145101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50559 (local address 127.0.0.1:36217)
0806 16:08:22.145226 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:22.145230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:22.145422 (+ 192us) server_negotiation.cc:407] Connection header received
0806 16:08:22.145552 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:22.145555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:22.145588 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:22.145680 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:22.146425 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:22.146428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:22.146431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:22.146442 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:22.146490 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:22.146503 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:22.146583 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:22.146678 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 16:08:22.146718 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 16:08:22.147061 11111 leader_election.cc:302] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27060d5e36a14d9bbf11e8d8ba0a8d25; no voters:
I20260806 16:08:22.147387 11111 leader_election.cc:288] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:22.147476 11122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:22.147697 11122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [term 1 LEADER]: Becoming Leader. State: Replica: 27060d5e36a14d9bbf11e8d8ba0a8d25, State: Running, Role: LEADER
I20260806 16:08:22.148044 11122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } }
I20260806 16:08:22.148598 11111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:22.149680 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27060d5e36a14d9bbf11e8d8ba0a8d25. Latest consensus state: current_term: 1 leader_uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } } }
I20260806 16:08:22.149806 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:22.150089 11123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27060d5e36a14d9bbf11e8d8ba0a8d25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36217 } } }
I20260806 16:08:22.150185 11123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:22.150209 11134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:22.153376 11134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:22.159368 11134 catalog_manager.cc:1347] Generated new cluster ID: ef81be4e986544f692309a17d401ab78
I20260806 16:08:22.159435 11134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:22.168401 11134 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:22.168682 11134 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:22.177237 11134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27060d5e36a14d9bbf11e8d8ba0a8d25: Generated new TSK 0
I20260806 16:08:22.178077 11134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:22.192211 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 11015
2026-08-06T16:08:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1674 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:22 dist-test-slave-7kzw krb5kdc[11164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:22 dist-test-slave-7kzw krb5kdc[11164](info): set up 2 sockets
Aug 06 16:08:22 dist-test-slave-7kzw krb5kdc[11164](info): commencing operation
krb5kdc: starting...
W20260806 16:08:23.398284 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:23 dist-test-slave-7kzw krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:23.534387 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:23.673454 11187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:23.673841 11187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:23.673931 11187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:23.678131 11187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:23.678231 11187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:23.678270 11187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:23.678308 11187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:23.678344 11187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:23.683835 11187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:23.684927 11187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:23.685976 11187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:23.694877 11194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:23.694837 11196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:23.694857 11193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:23.695076 11187 server_base.cc:1034] running on GCE node
I20260806 16:08:23.695623 11187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:23.696779 11187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:23.697968 11187 hybrid_clock.cc:648] HybridClock initialized: now 1786032503697948 us; error 35 us; skew 500 ppm
Aug 06 16:08:23 dist-test-slave-7kzw krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:23.700897 11187 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:23.701736 11187 webserver.cc:460] Webserver started at http://127.0.0.1:42837/ using document root <none> and password file <none>
I20260806 16:08:23.702255 11187 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:23.702320 11187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:23.702554 11187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:23.704555 11187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "3824258390af48f992c3766818e2d6ea"
format_stamp: "Formatted at 2026-08-06 16:08:23 on dist-test-slave-7kzw"
server_key: "cec9a04ff7c96918c6b59fc62b968e80"
server_key_iv: "01b10c3bfeb50553263523ed93c7441b"
server_key_version: "encryptionkey@0"
I20260806 16:08:23.705070 11187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "3824258390af48f992c3766818e2d6ea"
format_stamp: "Formatted at 2026-08-06 16:08:23 on dist-test-slave-7kzw"
server_key: "cec9a04ff7c96918c6b59fc62b968e80"
server_key_iv: "01b10c3bfeb50553263523ed93c7441b"
server_key_version: "encryptionkey@0"
I20260806 16:08:23.709091 11187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 16:08:23.711483 11205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:23.712694 11187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:23.712847 11187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "3824258390af48f992c3766818e2d6ea"
format_stamp: "Formatted at 2026-08-06 16:08:23 on dist-test-slave-7kzw"
server_key: "cec9a04ff7c96918c6b59fc62b968e80"
server_key_iv: "01b10c3bfeb50553263523ed93c7441b"
server_key_version: "encryptionkey@0"
I20260806 16:08:23.712956 11187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:23.733374 11187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:23.736714 11187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:23.736935 11187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:23.744611 11187 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46815
I20260806 16:08:23.744645 11276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46815 every 8 connection(s)
I20260806 16:08:23.745572 11187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:23.748620 11277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:23.752364 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 11187
I20260806 16:08:23.752537 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:23.752864 13904 external_mini_cluster.cc:1428] Setting key e4e38a65dde34332ec9fb5ec01bca4aa
I20260806 16:08:23.755194 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea: Bootstrap starting.
I20260806 16:08:23.757565 11277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:23.758358 11277 log.cc:824] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:23.760622 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea: No bootstrap required, opened a new log
Aug 06 16:08:23 dist-test-slave-7kzw krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 16:08:23.765767 11277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3824258390af48f992c3766818e2d6ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260806 16:08:23.766073 11277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:23.766145 11277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3824258390af48f992c3766818e2d6ea, State: Initialized, Role: FOLLOWER
I20260806 16:08:23.766724 11277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3824258390af48f992c3766818e2d6ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260806 16:08:23.766911 11277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:23.766996 11277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:23.767115 11277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:23.768357 11277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3824258390af48f992c3766818e2d6ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260806 16:08:23.768754 11277 leader_election.cc:302] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3824258390af48f992c3766818e2d6ea; no voters:
I20260806 16:08:23.769097 11277 leader_election.cc:288] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:23.769218 11282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:23.769470 11282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [term 1 LEADER]: Becoming Leader. State: Replica: 3824258390af48f992c3766818e2d6ea, State: Running, Role: LEADER
I20260806 16:08:23.769809 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:23.754399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:54285 (local address 127.0.0.1:46815)
0806 16:08:23.754899 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:23.754915 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:23.754953 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:08:23.755638 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:23.755656 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:23.755999 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:23.756332 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:23.757250 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:23.758583 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:23.759855 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:23.760322 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:23.760444 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:23.763777 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:23.763787 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:23.763809 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:23.763852 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:23.766064 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:23.766760 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:23.766766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:23.766772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:23.766931 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:23.767406 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:23.767410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:23.767412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:23.767938 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:23.768220 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:23.768483 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 16:08:23.768759 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":134,"threads_started":1}
I20260806 16:08:23.769860 11282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3824258390af48f992c3766818e2d6ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260806 16:08:23.770430 11277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:23.771507 11284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3824258390af48f992c3766818e2d6ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3824258390af48f992c3766818e2d6ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } } }
I20260806 16:08:23.771633 11284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:23.771960 11283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3824258390af48f992c3766818e2d6ea. Latest consensus state: current_term: 1 leader_uuid: "3824258390af48f992c3766818e2d6ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3824258390af48f992c3766818e2d6ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } } }
I20260806 16:08:23.772017 11293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:23.772046 11283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:23.773535 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:23.775828 11293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:23.782521 11293 catalog_manager.cc:1347] Generated new cluster ID: 0e7a52a79ef74a898f0ca885b6a8329a
I20260806 16:08:23.782650 11293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:23.787213 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:23.777199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:52553 (local address 127.0.0.1:46815)
0806 16:08:23.777308 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:23.777312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:23.777327 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:08:23.777364 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:23.777367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:23.777412 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:23.777489 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:23.778610 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:23.779435 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:23.780703 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:23.781010 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:23.781147 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:23.783157 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:23.783161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:23.783164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:23.783192 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:23.784952 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:23.785517 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:23.785520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:23.785522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:23.785604 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:23.785971 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:23.785974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:23.785975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:23.786190 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:23.786319 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:23.786951 (+ 632us) server_negotiation.cc:299] Negotiation successful
0806 16:08:23.787043 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:08:23.797847 11293 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:23.799069 11293 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:23.827533 11293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3824258390af48f992c3766818e2d6ea: Generated new TSK 0
I20260806 16:08:23.828454 11293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:23.834271 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 11187
W20260806 16:08:23.845803 11162 connection.cc:535] client connection to 127.0.0.1:46815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46815): Connection reset by peer (error 104)
2026-08-06T16:08:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1640 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:23 dist-test-slave-7kzw krb5kdc[11322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:23 dist-test-slave-7kzw krb5kdc[11322](info): set up 2 sockets
Aug 06 16:08:23 dist-test-slave-7kzw krb5kdc[11322](info): commencing operation
krb5kdc: starting...
W20260806 16:08:25.097468 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:25 dist-test-slave-7kzw krb5kdc[11322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:25.233810 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:25.346871 11342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:25.347193 11342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:25.347297 11342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:25.351354 11342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:25.351442 11342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:25.351480 11342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:25.351533 11342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:25.351574 11342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:25.356807 11342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:25.357874 11342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:25.359005 11342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:25.367681 11351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:25.367699 11350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:25.367715 11353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:25.367738 11342 server_base.cc:1034] running on GCE node
I20260806 16:08:25.368572 11342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:25.369704 11342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:25.370869 11342 hybrid_clock.cc:648] HybridClock initialized: now 1786032505370834 us; error 51 us; skew 500 ppm
Aug 06 16:08:25 dist-test-slave-7kzw krb5kdc[11322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:25.373853 11342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:25.374790 11342 webserver.cc:460] Webserver started at http://127.0.0.1:39337/ using document root <none> and password file <none>
I20260806 16:08:25.375358 11342 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:25.375452 11342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:25.375702 11342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:25.377763 11342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "578944fa89094aa3babb19165d628050"
format_stamp: "Formatted at 2026-08-06 16:08:25 on dist-test-slave-7kzw"
server_key: "3079e7812872613f209cc9a64429ce7d"
server_key_iv: "8c91b06563035e6f793c9237adf9de04"
server_key_version: "encryptionkey@0"
I20260806 16:08:25.378337 11342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "578944fa89094aa3babb19165d628050"
format_stamp: "Formatted at 2026-08-06 16:08:25 on dist-test-slave-7kzw"
server_key: "3079e7812872613f209cc9a64429ce7d"
server_key_iv: "8c91b06563035e6f793c9237adf9de04"
server_key_version: "encryptionkey@0"
I20260806 16:08:25.382114 11342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:08:25.384534 11361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:25.385849 11342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:25.386023 11342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "578944fa89094aa3babb19165d628050"
format_stamp: "Formatted at 2026-08-06 16:08:25 on dist-test-slave-7kzw"
server_key: "3079e7812872613f209cc9a64429ce7d"
server_key_iv: "8c91b06563035e6f793c9237adf9de04"
server_key_version: "encryptionkey@0"
I20260806 16:08:25.386148 11342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:25.399290 11342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:25.402487 11342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:25.402722 11342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:25.409898 11342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36785
I20260806 16:08:25.409921 11425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36785 every 8 connection(s)
I20260806 16:08:25.410903 11342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:25.413574 11426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:25.419025 11426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050: Bootstrap starting.
I20260806 16:08:25.421043 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 11342
I20260806 16:08:25.421201 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:25.421463 11426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:25.421486 13904 external_mini_cluster.cc:1428] Setting key 1a53cdab02584b150ab6e38c6e03e457
I20260806 16:08:25.422199 11426 log.cc:824] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:25.424587 11426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050: No bootstrap required, opened a new log
I20260806 16:08:25.429040 11426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578944fa89094aa3babb19165d628050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36785 } }
I20260806 16:08:25.429375 11426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:25.429466 11426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578944fa89094aa3babb19165d628050, State: Initialized, Role: FOLLOWER
I20260806 16:08:25.429903 11426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578944fa89094aa3babb19165d628050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36785 } }
I20260806 16:08:25.430079 11426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:25.430172 11426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:25.430272 11426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:25.431371 11426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578944fa89094aa3babb19165d628050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36785 } }
I20260806 16:08:25.431470 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:25.422664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:35575 (local address 127.0.0.1:36785)
0806 16:08:25.423178 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:25.423200 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:25.423246 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:08:25.424110 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:25.424133 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:25.424542 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:25.424897 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:25.425324 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:25.426801 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:25.427757 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:25.428465 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:25.428637 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:25.429635 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:25.429640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:25.429652 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:25.430129 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:25.430241 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:25.430277 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:25.430428 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:25.430643 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 16:08:25.430773 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":140,"threads_started":1}
I20260806 16:08:25.431789 11426 leader_election.cc:302] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 578944fa89094aa3babb19165d628050; no voters:
I20260806 16:08:25.432123 11426 leader_election.cc:288] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:25.432228 11431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:25.432478 11431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [term 1 LEADER]: Becoming Leader. State: Replica: 578944fa89094aa3babb19165d628050, State: Running, Role: LEADER
I20260806 16:08:25.433192 11426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:25.433503 11431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578944fa89094aa3babb19165d628050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36785 } }
I20260806 16:08:25.434608 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:25.435869 11433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 578944fa89094aa3babb19165d628050. Latest consensus state: current_term: 1 leader_uuid: "578944fa89094aa3babb19165d628050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578944fa89094aa3babb19165d628050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36785 } } }
I20260806 16:08:25.436501 11433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:25.436208 11432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "578944fa89094aa3babb19165d628050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "578944fa89094aa3babb19165d628050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36785 } } }
I20260806 16:08:25.436753 11432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050 [sys.catalog]: This master's current role is: LEADER
W20260806 16:08:25.438738 11447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:25.438822 11447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:25.438890 11456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:25.439615 11456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:25.441663 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:25.436916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:42525 (local address 127.0.0.1:36785)
0806 16:08:25.437006 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:25.437010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:25.437661 (+ 651us) server_negotiation.cc:407] Connection header received
0806 16:08:25.437846 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:25.437849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:25.437894 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:25.437950 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:25.438383 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:25.439064 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:25.439952 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:25.440198 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:25.440317 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:25.441180 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:25.441184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:25.441186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:25.441199 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:25.441232 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:25.441244 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:25.441341 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:25.441464 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 16:08:25.441504 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 16:08:25.445303 11456 catalog_manager.cc:1347] Generated new cluster ID: 0630fa1b853d477c9456261e6c0d6723
I20260806 16:08:25.445369 11456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:25.461632 11456 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:25.462630 11456 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:25.476454 11456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 578944fa89094aa3babb19165d628050: Generated new TSK 0
I20260806 16:08:25.477196 11456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:25.491055 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 11342
W20260806 16:08:25.504352 11319 connection.cc:535] client connection to 127.0.0.1:36785 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36785): Connection reset by peer (error 104)
2026-08-06T16:08:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1658 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:25 dist-test-slave-7kzw krb5kdc[11465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:25 dist-test-slave-7kzw krb5kdc[11465](info): set up 2 sockets
Aug 06 16:08:25 dist-test-slave-7kzw krb5kdc[11465](info): commencing operation
krb5kdc: starting...
W20260806 16:08:26.761370 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:26 dist-test-slave-7kzw krb5kdc[11465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:26.903129 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:27.017786 11483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:27.018049 11483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:27.018090 11483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:27.022264 11483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:27.022332 11483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:27.022352 11483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:27.022369 11483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:27.022384 11483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:27.027992 11483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:27.029145 11483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:27.030236 11483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:27.039366 11496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:27.039355 11494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:27.039355 11493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:27.039813 11483 server_base.cc:1034] running on GCE node
I20260806 16:08:27.040452 11483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:27.041654 11483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:27.042845 11483 hybrid_clock.cc:648] HybridClock initialized: now 1786032507042810 us; error 49 us; skew 500 ppm
Aug 06 16:08:27 dist-test-slave-7kzw krb5kdc[11465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:27.045941 11483 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:27.046984 11483 webserver.cc:460] Webserver started at http://127.0.0.1:42707/ using document root <none> and password file <none>
I20260806 16:08:27.047622 11483 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:27.047726 11483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:27.047972 11483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:27.050084 11483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076"
format_stamp: "Formatted at 2026-08-06 16:08:27 on dist-test-slave-7kzw"
server_key: "91d24696c1be1f976979d2e8f4d734f9"
server_key_iv: "79bd21ca563178f17f609b02abc69921"
server_key_version: "encryptionkey@0"
I20260806 16:08:27.050738 11483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076"
format_stamp: "Formatted at 2026-08-06 16:08:27 on dist-test-slave-7kzw"
server_key: "91d24696c1be1f976979d2e8f4d734f9"
server_key_iv: "79bd21ca563178f17f609b02abc69921"
server_key_version: "encryptionkey@0"
I20260806 16:08:27.054531 11483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:08:27.057176 11506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:27.058456 11483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:27.058707 11483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076"
format_stamp: "Formatted at 2026-08-06 16:08:27 on dist-test-slave-7kzw"
server_key: "91d24696c1be1f976979d2e8f4d734f9"
server_key_iv: "79bd21ca563178f17f609b02abc69921"
server_key_version: "encryptionkey@0"
I20260806 16:08:27.058827 11483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:27.063526 11483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:27.066903 11483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:27.067107 11483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:27.074179 11483 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42987
I20260806 16:08:27.074200 11585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42987 every 8 connection(s)
I20260806 16:08:27.075150 11483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:27.077986 11586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:27.080786 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 11483
I20260806 16:08:27.080942 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:27.081248 13904 external_mini_cluster.cc:1428] Setting key bbf86cbceb9435bd4353f8c2defd1ed3
I20260806 16:08:27.084057 11586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076: Bootstrap starting.
I20260806 16:08:27.086596 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:27.082742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:55633 (local address 127.0.0.1:42987)
0806 16:08:27.083209 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:27.083225 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:27.083260 (+ 35us) server_negotiation.cc:407] Connection header received
0806 16:08:27.083912 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:27.083930 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:27.084249 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:27.084590 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:27.085370 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:27.085375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:27.085386 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:27.085396 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:27.085449 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:27.085468 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:27.085629 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:27.085701 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 16:08:27.085856 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":139,"threads_started":1}
I20260806 16:08:27.086601 11586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:27.087431 11586 log.cc:824] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:27.089497 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:27.089560 11586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076: No bootstrap required, opened a new log
I20260806 16:08:27.093370 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:27.091627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:60845 (local address 127.0.0.1:42987)
0806 16:08:27.091753 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:27.091758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:27.091893 (+ 135us) server_negotiation.cc:407] Connection header received
0806 16:08:27.092035 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:27.092038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:27.092087 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:27.092173 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:27.092930 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:27.092934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:27.092936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:27.092947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:27.092980 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:27.092990 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:27.093081 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:27.093212 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 16:08:27.093251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:08:27.093708 11586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } }
I20260806 16:08:27.094002 11586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:27.094086 11586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc3750c47ab4c9890ae2a3ddd1a0076, State: Initialized, Role: FOLLOWER
I20260806 16:08:27.094585 11586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } }
I20260806 16:08:27.094746 11586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:27.094815 11586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:27.094913 11586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:27.095806 11586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } }
I20260806 16:08:27.096155 11586 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc3750c47ab4c9890ae2a3ddd1a0076; no voters:
I20260806 16:08:27.096458 11586 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:27.096617 11600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:27.096848 11600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [term 1 LEADER]: Becoming Leader. State: Replica: 2bc3750c47ab4c9890ae2a3ddd1a0076, State: Running, Role: LEADER
I20260806 16:08:27.097218 11600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } }
I20260806 16:08:27.097705 11586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:27.098909 11602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc3750c47ab4c9890ae2a3ddd1a0076. Latest consensus state: current_term: 1 leader_uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } } }
I20260806 16:08:27.099045 11602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:27.099436 11610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:27.100167 11601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc3750c47ab4c9890ae2a3ddd1a0076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } } }
I20260806 16:08:27.100301 11601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:27.102624 11610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:27.108537 11610 catalog_manager.cc:1347] Generated new cluster ID: 4160ee052531433898a003fe4450807d
I20260806 16:08:27.108624 11610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:27.120456 11610 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:27.120793 11610 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:27.126520 11610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bc3750c47ab4c9890ae2a3ddd1a0076: Generated new TSK 0
I20260806 16:08:27.127332 11610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:27.138290 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 11483
2026-08-06T16:08:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1646 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:27 dist-test-slave-7kzw krb5kdc[11622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:27 dist-test-slave-7kzw krb5kdc[11622](info): set up 2 sockets
Aug 06 16:08:27 dist-test-slave-7kzw krb5kdc[11622](info): commencing operation
krb5kdc: starting...
W20260806 16:08:28.350116 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:28 dist-test-slave-7kzw krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:28.484994 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:28.597142 11642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:28.597414 11642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:28.597455 11642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:28.601305 11642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:28.601358 11642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:28.601377 11642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:28.601393 11642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:28.601408 11642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:28.606490 11642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:28.607527 11642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:28.608515 11642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:28.617673 11654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:28.617694 11648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:28.617695 11650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:28.618016 11642 server_base.cc:1034] running on GCE node
I20260806 16:08:28.618914 11642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:28.620105 11642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:28.621295 11642 hybrid_clock.cc:648] HybridClock initialized: now 1786032508621267 us; error 40 us; skew 500 ppm
Aug 06 16:08:28 dist-test-slave-7kzw krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:28.630810 11642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:28.631776 11642 webserver.cc:460] Webserver started at http://127.0.0.1:35391/ using document root <none> and password file <none>
I20260806 16:08:28.632472 11642 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:28.632546 11642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:28.632778 11642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:28.634927 11642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e"
format_stamp: "Formatted at 2026-08-06 16:08:28 on dist-test-slave-7kzw"
server_key: "83d5bcf1c5f3f13d398450d221bc0ed2"
server_key_iv: "5c59427a9bb431130b8f265f88ecc055"
server_key_version: "encryptionkey@0"
I20260806 16:08:28.635522 11642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e"
format_stamp: "Formatted at 2026-08-06 16:08:28 on dist-test-slave-7kzw"
server_key: "83d5bcf1c5f3f13d398450d221bc0ed2"
server_key_iv: "5c59427a9bb431130b8f265f88ecc055"
server_key_version: "encryptionkey@0"
I20260806 16:08:28.639422 11642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:08:28.641991 11664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:28.643410 11642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:08:28.643573 11642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e"
format_stamp: "Formatted at 2026-08-06 16:08:28 on dist-test-slave-7kzw"
server_key: "83d5bcf1c5f3f13d398450d221bc0ed2"
server_key_iv: "5c59427a9bb431130b8f265f88ecc055"
server_key_version: "encryptionkey@0"
I20260806 16:08:28.643692 11642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:28.659269 11642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:28.662849 11642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:28.663067 11642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:28.670456 11642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44815
I20260806 16:08:28.670509 11733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44815 every 8 connection(s)
I20260806 16:08:28.671489 11642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:28.674225 11734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:28.679719 11734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e: Bootstrap starting.
I20260806 16:08:28.682021 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 11642
I20260806 16:08:28.682155 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:28.682334 11734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:28.682468 13904 external_mini_cluster.cc:1428] Setting key a9ff96dbefd9db1713ae7af80b9624f8
I20260806 16:08:28.683256 11734 log.cc:824] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:28.685539 11734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e: No bootstrap required, opened a new log
I20260806 16:08:28.689399 11734 raft_consensus.cc:348] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260806 16:08:28.689672 11734 raft_consensus.cc:374] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:28.689745 11734 raft_consensus.cc:729] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35d0b6c77104fe2abb2c8bb929bbd6e, State: Initialized, Role: FOLLOWER
I20260806 16:08:28.690655 11734 consensus_queue.cc:260] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260806 16:08:28.690831 11734 raft_consensus.cc:388] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:28.690930 11734 raft_consensus.cc:482] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:28.691064 11734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 0 FOLLOWER]: Advancing to term 1
Aug 06 16:08:28 dist-test-slave-7kzw krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 16:08:28.692417 11734 raft_consensus.cc:504] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260806 16:08:28.692796 11734 leader_election.cc:302] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e35d0b6c77104fe2abb2c8bb929bbd6e; no voters:
I20260806 16:08:28.693207 11734 leader_election.cc:288] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:28.693338 11740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:28.693590 11740 raft_consensus.cc:686] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [term 1 LEADER]: Becoming Leader. State: Replica: e35d0b6c77104fe2abb2c8bb929bbd6e, State: Running, Role: LEADER
I20260806 16:08:28.693974 11740 consensus_queue.cc:237] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260806 16:08:28.694747 11734 sys_catalog.cc:564] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:28.695730 11741 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } } }
I20260806 16:08:28.696138 11741 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:28.696027 11747 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e35d0b6c77104fe2abb2c8bb929bbd6e. Latest consensus state: current_term: 1 leader_uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35d0b6c77104fe2abb2c8bb929bbd6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } } }
I20260806 16:08:28.696457 11747 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:28.697388 11758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:28.699558 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:28.683739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:34395 (local address 127.0.0.1:44815)
0806 16:08:28.684234 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:28.684255 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:28.684303 (+ 48us) server_negotiation.cc:407] Connection header received
0806 16:08:28.685001 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:28.685019 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:28.685379 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:28.685779 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:28.686761 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:28.687855 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:28.688829 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:28.689374 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:28.689594 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:28.692697 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:28.692702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:28.692716 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:28.692749 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:28.695003 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:28.695745 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:28.695752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:28.695759 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:28.695940 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:28.696467 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:28.696471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:28.696473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:28.697091 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:28.697348 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:28.698219 (+ 871us) server_negotiation.cc:299] Negotiation successful
0806 16:08:28.698498 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":169,"threads_started":1}
I20260806 16:08:28.700991 11758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:28.703107 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:28.707352 11758 catalog_manager.cc:1347] Generated new cluster ID: 1b43788c09f2410eb7b640c554f1c31c
I20260806 16:08:28.707417 11758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:28.714895 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:28.706014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:56011 (local address 127.0.0.1:44815)
0806 16:08:28.706148 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:28.706152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:28.706169 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:08:28.706206 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:28.706208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:28.706261 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:28.706345 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:28.707343 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:28.708027 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:28.708916 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:28.709197 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:28.709322 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:28.711026 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:28.711031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:28.711034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:28.711065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:28.712792 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:28.713344 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:28.713348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:28.713351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:28.713444 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:28.713828 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:28.713832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:28.713834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:28.714033 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:28.714222 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:28.714556 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 16:08:28.714681 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:08:28.723835 11758 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:28.724900 11758 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:28.742537 11758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e35d0b6c77104fe2abb2c8bb929bbd6e: Generated new TSK 0
I20260806 16:08:28.743471 11758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:28.752671 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 11642
W20260806 16:08:28.765388 11620 connection.cc:535] client connection to 127.0.0.1:44815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44815): Connection reset by peer (error 104)
2026-08-06T16:08:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:28 dist-test-slave-7kzw krb5kdc[11782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:28 dist-test-slave-7kzw krb5kdc[11782](info): set up 2 sockets
Aug 06 16:08:28 dist-test-slave-7kzw krb5kdc[11782](info): commencing operation
krb5kdc: starting...
W20260806 16:08:29.972265 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:30 dist-test-slave-7kzw krb5kdc[11782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:30.113862 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:30.231179 11801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:30.231458 11801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:30.231503 11801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:30.235654 11801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:30.235737 11801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:30.235759 11801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:30.235775 11801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:30.235791 11801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:30.241406 11801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:30.242635 11801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:30.243736 11801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:30.252611 11809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:30.252743 11801 server_base.cc:1034] running on GCE node
W20260806 16:08:30.252650 11807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:30.252568 11811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:30.253480 11801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:30.254719 11801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:30.255926 11801 hybrid_clock.cc:648] HybridClock initialized: now 1786032510255906 us; error 37 us; skew 500 ppm
Aug 06 16:08:30 dist-test-slave-7kzw krb5kdc[11782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:30.259019 11801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:30.260030 11801 webserver.cc:460] Webserver started at http://127.0.0.1:45637/ using document root <none> and password file <none>
I20260806 16:08:30.260634 11801 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:30.260735 11801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:30.260985 11801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:30.263334 11801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "edbe3c73db27413aa8a4bb050f8ee9db"
format_stamp: "Formatted at 2026-08-06 16:08:30 on dist-test-slave-7kzw"
server_key: "25fde75d2ed038ab9c9144f7da004621"
server_key_iv: "4d73d012e4b62cd9d7d6124eb3e0b188"
server_key_version: "encryptionkey@0"
I20260806 16:08:30.263942 11801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "edbe3c73db27413aa8a4bb050f8ee9db"
format_stamp: "Formatted at 2026-08-06 16:08:30 on dist-test-slave-7kzw"
server_key: "25fde75d2ed038ab9c9144f7da004621"
server_key_iv: "4d73d012e4b62cd9d7d6124eb3e0b188"
server_key_version: "encryptionkey@0"
I20260806 16:08:30.268502 11801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:08:30.271142 11826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:30.272437 11801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:30.272588 11801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "edbe3c73db27413aa8a4bb050f8ee9db"
format_stamp: "Formatted at 2026-08-06 16:08:30 on dist-test-slave-7kzw"
server_key: "25fde75d2ed038ab9c9144f7da004621"
server_key_iv: "4d73d012e4b62cd9d7d6124eb3e0b188"
server_key_version: "encryptionkey@0"
I20260806 16:08:30.272734 11801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:30.290118 11801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:30.293303 11801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:30.293517 11801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:30.301133 11801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37073
I20260806 16:08:30.301189 11892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37073 every 8 connection(s)
I20260806 16:08:30.302094 11801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:30.304806 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:30.310325 11893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db: Bootstrap starting.
I20260806 16:08:30.312189 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 11801
I20260806 16:08:30.312340 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:30.312647 13904 external_mini_cluster.cc:1428] Setting key 0fd7cd7704fa1281b6bb6eddf02a6c0b
I20260806 16:08:30.313123 11893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:30.313997 11893 log.cc:824] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:30.315989 11893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db: No bootstrap required, opened a new log
I20260806 16:08:30.319631 11893 raft_consensus.cc:348] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbe3c73db27413aa8a4bb050f8ee9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260806 16:08:30.319896 11893 raft_consensus.cc:374] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:30.319980 11893 raft_consensus.cc:729] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbe3c73db27413aa8a4bb050f8ee9db, State: Initialized, Role: FOLLOWER
I20260806 16:08:30.320452 11893 consensus_queue.cc:260] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbe3c73db27413aa8a4bb050f8ee9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260806 16:08:30.320601 11893 raft_consensus.cc:388] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:30.320676 11893 raft_consensus.cc:482] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:30.320808 11893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:30.321718 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:30.313849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:37253 (local address 127.0.0.1:37073)
0806 16:08:30.314335 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:30.314378 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:30.314425 (+ 47us) server_negotiation.cc:407] Connection header received
0806 16:08:30.315266 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:30.315286 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:30.315612 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:30.315947 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:30.316364 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:30.317295 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:30.318186 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:30.318739 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:30.318917 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:30.319923 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:30.319930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:30.319949 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:30.320401 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:30.320454 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:30.320478 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:30.320625 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:30.320827 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 16:08:30.320969 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":136,"threads_started":1}
I20260806 16:08:30.321753 11893 raft_consensus.cc:504] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbe3c73db27413aa8a4bb050f8ee9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260806 16:08:30.322134 11893 leader_election.cc:302] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edbe3c73db27413aa8a4bb050f8ee9db; no voters:
I20260806 16:08:30.322583 11893 leader_election.cc:288] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:30.322680 11901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:30.323084 11901 raft_consensus.cc:686] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [term 1 LEADER]: Becoming Leader. State: Replica: edbe3c73db27413aa8a4bb050f8ee9db, State: Running, Role: LEADER
I20260806 16:08:30.323400 11901 consensus_queue.cc:237] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbe3c73db27413aa8a4bb050f8ee9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260806 16:08:30.324066 11893 sys_catalog.cc:564] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:30.325083 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:30.325570 11904 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edbe3c73db27413aa8a4bb050f8ee9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbe3c73db27413aa8a4bb050f8ee9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } } }
I20260806 16:08:30.325695 11904 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:30.326431 11905 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [sys.catalog]: SysCatalogTable state changed. Reason: New leader edbe3c73db27413aa8a4bb050f8ee9db. Latest consensus state: current_term: 1 leader_uuid: "edbe3c73db27413aa8a4bb050f8ee9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbe3c73db27413aa8a4bb050f8ee9db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } } }
I20260806 16:08:30.326574 11905 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db [sys.catalog]: This master's current role is: LEADER
W20260806 16:08:30.329682 11927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:30.329774 11927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:30.329851 11916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:30.330627 11916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:30.332387 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:30.327430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:47367 (local address 127.0.0.1:37073)
0806 16:08:30.327541 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:30.327545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:30.328013 (+ 468us) server_negotiation.cc:407] Connection header received
0806 16:08:30.328101 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:30.328105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:30.328152 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:30.328238 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:30.328562 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:30.329298 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:30.330446 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:30.330806 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:30.330920 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:30.331887 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:30.331891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:30.331894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:30.331907 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:30.331938 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:30.331952 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:30.332046 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:30.332226 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 16:08:30.332264 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 16:08:30.336084 11916 catalog_manager.cc:1347] Generated new cluster ID: b2c53c3e59c74ccaabf1aa5879b3620d
I20260806 16:08:30.336158 11916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:30.349196 11916 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:30.350209 11916 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:30.356549 11916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edbe3c73db27413aa8a4bb050f8ee9db: Generated new TSK 0
I20260806 16:08:30.357291 11916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:30.363440 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 11801
W20260806 16:08:30.377169 11780 connection.cc:535] client connection to 127.0.0.1:37073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37073): Connection reset by peer (error 104)
2026-08-06T16:08:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1611 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:30 dist-test-slave-7kzw krb5kdc[11940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:30 dist-test-slave-7kzw krb5kdc[11940](info): set up 2 sockets
Aug 06 16:08:30 dist-test-slave-7kzw krb5kdc[11940](info): commencing operation
krb5kdc: starting...
W20260806 16:08:31.597244 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:31 dist-test-slave-7kzw krb5kdc[11940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 16:08:31.733911 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:31.847293 11958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:31.847594 11958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:31.847668 11958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:31.851694 11958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:31.851786 11958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:31.851824 11958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:31.851884 11958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:31.851923 11958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:31.857117 11958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:31.858143 11958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:31.859212 11958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:31.867822 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:31.867869 11958 server_base.cc:1034] running on GCE node
W20260806 16:08:31.867810 11967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:31.867794 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:31.868593 11958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:31.869690 11958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:31.870865 11958 hybrid_clock.cc:648] HybridClock initialized: now 1786032511870837 us; error 42 us; skew 500 ppm
Aug 06 16:08:31 dist-test-slave-7kzw krb5kdc[11940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:31.873641 11958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 16:08:31.874595 11958 webserver.cc:460] Webserver started at http://127.0.0.1:34307/ using document root <none> and password file <none>
I20260806 16:08:31.875167 11958 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:31.875264 11958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:31.875504 11958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:31.877595 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "3e7cea86519844509d361fa238285ac0"
format_stamp: "Formatted at 2026-08-06 16:08:31 on dist-test-slave-7kzw"
server_key: "aea046f70ae37d604e46d2bba3864822"
server_key_iv: "d101483f8507b2b3fa10c67ce7548efc"
server_key_version: "encryptionkey@0"
I20260806 16:08:31.878187 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "3e7cea86519844509d361fa238285ac0"
format_stamp: "Formatted at 2026-08-06 16:08:31 on dist-test-slave-7kzw"
server_key: "aea046f70ae37d604e46d2bba3864822"
server_key_iv: "d101483f8507b2b3fa10c67ce7548efc"
server_key_version: "encryptionkey@0"
I20260806 16:08:31.881953 11958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:08:31.884549 11977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:31.885865 11958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:31.886026 11958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "3e7cea86519844509d361fa238285ac0"
format_stamp: "Formatted at 2026-08-06 16:08:31 on dist-test-slave-7kzw"
server_key: "aea046f70ae37d604e46d2bba3864822"
server_key_iv: "d101483f8507b2b3fa10c67ce7548efc"
server_key_version: "encryptionkey@0"
I20260806 16:08:31.886145 11958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:31.891542 11958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:31.894836 11958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:31.894986 11958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:31.901932 11958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42577
I20260806 16:08:31.902055 12043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42577 every 8 connection(s)
I20260806 16:08:31.902971 11958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:31.905897 12044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:31.910830 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 11958
I20260806 16:08:31.910989 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:31.911298 13904 external_mini_cluster.cc:1428] Setting key 848a6cdd20c9574a646cf89189ac6208
I20260806 16:08:31.912655 12044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0: Bootstrap starting.
I20260806 16:08:31.915365 12044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:31.916203 12044 log.cc:824] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:31.917397 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:31.913003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:46693 (local address 127.0.0.1:42577)
0806 16:08:31.913515 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:31.913530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:31.913566 (+ 36us) server_negotiation.cc:407] Connection header received
0806 16:08:31.914226 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:31.914243 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:31.914603 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:31.914958 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:31.915887 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:31.915892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:31.915903 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:31.915913 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:31.915962 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:31.915982 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:31.916113 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:31.916253 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 16:08:31.916437 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":165,"threads_started":1}
I20260806 16:08:31.918731 12044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0: No bootstrap required, opened a new log
I20260806 16:08:31.920567 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:31.923256 12044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7cea86519844509d361fa238285ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42577 } }
I20260806 16:08:31.923609 12044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:31.923681 12044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7cea86519844509d361fa238285ac0, State: Initialized, Role: FOLLOWER
I20260806 16:08:31.924239 12044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7cea86519844509d361fa238285ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42577 } }
I20260806 16:08:31.924423 12044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:31.924528 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:31.922748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.136:38323 (local address 127.0.0.1:42577)
0806 16:08:31.922888 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:31.922893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:31.923042 (+ 149us) server_negotiation.cc:407] Connection header received
0806 16:08:31.923127 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:31.923130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:31.923170 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:31.923267 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:31.924051 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:31.924055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:31.924058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:31.924081 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:31.924115 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:31.924130 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:31.924227 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:31.924315 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 16:08:31.924359 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:08:31.924512 12044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:31.924662 12044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:31.925753 12044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7cea86519844509d361fa238285ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42577 } }
I20260806 16:08:31.926131 12044 leader_election.cc:302] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e7cea86519844509d361fa238285ac0; no voters:
I20260806 16:08:31.926518 12044 leader_election.cc:288] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:31.926617 12059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:31.926872 12059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [term 1 LEADER]: Becoming Leader. State: Replica: 3e7cea86519844509d361fa238285ac0, State: Running, Role: LEADER
I20260806 16:08:31.927186 12059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7cea86519844509d361fa238285ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42577 } }
I20260806 16:08:31.927938 12044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:31.928939 12061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e7cea86519844509d361fa238285ac0. Latest consensus state: current_term: 1 leader_uuid: "3e7cea86519844509d361fa238285ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7cea86519844509d361fa238285ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42577 } } }
I20260806 16:08:31.929207 12060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e7cea86519844509d361fa238285ac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7cea86519844509d361fa238285ac0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42577 } } }
I20260806 16:08:31.929319 12061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:31.929368 12060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:31.930310 12069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:31.933178 12069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:31.938726 12069 catalog_manager.cc:1347] Generated new cluster ID: 6ae53937c0c1494c88ee887c8f627cd1
I20260806 16:08:31.938815 12069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:31.984069 12069 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:31.984464 12069 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:32.004454 12069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e7cea86519844509d361fa238285ac0: Generated new TSK 0
I20260806 16:08:32.005360 12069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:32.021487 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 11958
2026-08-06T16:08:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1661 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:32 dist-test-slave-7kzw krb5kdc[12085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:32 dist-test-slave-7kzw krb5kdc[12085](info): set up 2 sockets
Aug 06 16:08:32 dist-test-slave-7kzw krb5kdc[12085](info): commencing operation
krb5kdc: starting...
W20260806 16:08:33.327199 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:33 dist-test-slave-7kzw krb5kdc[12085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:33.462226 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:37839
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:37839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:33.579043 12103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:33.579347 12103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:33.579434 12103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:33.583498 12103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:33.583591 12103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:33.583629 12103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:33.583688 12103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:33.583727 12103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:33.588914 12103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:37839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:37839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:33.589963 12103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:33.591063 12103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:33.599578 12112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:33.599642 12114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:33.599637 12116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:33.599697 12103 server_base.cc:1034] running on GCE node
I20260806 16:08:33.600401 12103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:33.601475 12103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:33.602696 12103 hybrid_clock.cc:648] HybridClock initialized: now 1786032513602675 us; error 34 us; skew 500 ppm
Aug 06 16:08:33 dist-test-slave-7kzw krb5kdc[12085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032513, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:33.605489 12103 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:33.606527 12103 webserver.cc:460] Webserver started at http://127.13.148.62:42113/ using document root <none> and password file <none>
I20260806 16:08:33.607089 12103 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:33.607194 12103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:33.607447 12103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:33.609596 12103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "bb5807d137fc42b289b763ba00da8e86"
format_stamp: "Formatted at 2026-08-06 16:08:33 on dist-test-slave-7kzw"
server_key: "bc4fb56d637ddb22cccebfc22e2a2ebe"
server_key_iv: "87f4270192c076ff3d121ffd618e2ec0"
server_key_version: "encryptionkey@0"
I20260806 16:08:33.610220 12103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "bb5807d137fc42b289b763ba00da8e86"
format_stamp: "Formatted at 2026-08-06 16:08:33 on dist-test-slave-7kzw"
server_key: "bc4fb56d637ddb22cccebfc22e2a2ebe"
server_key_iv: "87f4270192c076ff3d121ffd618e2ec0"
server_key_version: "encryptionkey@0"
I20260806 16:08:33.614033 12103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 16:08:33.616499 12126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:33.617769 12103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 16:08:33.617915 12103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "bb5807d137fc42b289b763ba00da8e86"
format_stamp: "Formatted at 2026-08-06 16:08:33 on dist-test-slave-7kzw"
server_key: "bc4fb56d637ddb22cccebfc22e2a2ebe"
server_key_iv: "87f4270192c076ff3d121ffd618e2ec0"
server_key_version: "encryptionkey@0"
I20260806 16:08:33.618034 12103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:33.639386 12103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:33.643029 12103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:33.643263 12103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:33.651459 12103 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:37839
I20260806 16:08:33.651490 12196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:37839 every 8 connection(s)
I20260806 16:08:33.652501 12103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:33.655937 12197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:33.659682 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 12103
I20260806 16:08:33.659835 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:33.660130 13904 external_mini_cluster.cc:1428] Setting key 96659f474957f108e6e495e804000494
I20260806 16:08:33.662897 12197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86: Bootstrap starting.
I20260806 16:08:33.666635 12197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:33.667654 12197 log.cc:824] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:33.669992 12197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86: No bootstrap required, opened a new log
I20260806 16:08:33.674705 12197 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5807d137fc42b289b763ba00da8e86" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37839 } }
I20260806 16:08:33.674993 12197 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:33.675071 12197 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5807d137fc42b289b763ba00da8e86, State: Initialized, Role: FOLLOWER
I20260806 16:08:33.675580 12197 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5807d137fc42b289b763ba00da8e86" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37839 } }
I20260806 16:08:33.675746 12197 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:33.675818 12197 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:33.675935 12197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:33.677251 12197 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5807d137fc42b289b763ba00da8e86" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37839 } }
I20260806 16:08:33.677626 12197 leader_election.cc:302] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb5807d137fc42b289b763ba00da8e86; no voters:
Aug 06 16:08:33 dist-test-slave-7kzw krb5kdc[12085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:33.677968 12197 leader_election.cc:288] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:33.678074 12206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:33.678361 12206 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [term 1 LEADER]: Becoming Leader. State: Replica: bb5807d137fc42b289b763ba00da8e86, State: Running, Role: LEADER
I20260806 16:08:33.678712 12206 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5807d137fc42b289b763ba00da8e86" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37839 } }
I20260806 16:08:33.679423 12197 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:33.680776 12207 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb5807d137fc42b289b763ba00da8e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5807d137fc42b289b763ba00da8e86" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37839 } } }
I20260806 16:08:33.680948 12207 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:33.681272 12207 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5807d137fc42b289b763ba00da8e86. Latest consensus state: current_term: 1 leader_uuid: "bb5807d137fc42b289b763ba00da8e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5807d137fc42b289b763ba00da8e86" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37839 } } }
I20260806 16:08:33.681386 12207 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:33.685971 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:33.666890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40153 (local address 127.13.148.62:37839)
0806 16:08:33.667453 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:33.667473 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:33.667519 (+ 46us) server_negotiation.cc:407] Connection header received
0806 16:08:33.668724 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:33.668761 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:33.669216 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:33.670762 (+ 1546us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:33.671205 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:33.672727 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:33.674145 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:33.674861 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:33.678807 (+ 3946us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:33.679228 (+ 421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:33.679234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:33.679251 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:33.679280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:33.682228 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:33.682922 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:33.682926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:33.682931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:33.683054 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:33.683480 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:33.683484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:33.683487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:33.684153 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:33.684469 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:33.684856 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 16:08:33.685129 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":165,"threads_started":1}
W20260806 16:08:33.687037 12226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:33.687124 12226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:33.687214 12220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:33.688197 12220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:33.689428 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:33.694686 12220 catalog_manager.cc:1347] Generated new cluster ID: 50f24035bdb1431a81d69885176b24be
I20260806 16:08:33.694769 12220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:33.703783 12220 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:33.705068 12220 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:33.707736 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:33.693386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43639 (local address 127.13.148.62:37839)
0806 16:08:33.693537 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:33.693544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:33.695220 (+ 1676us) server_negotiation.cc:407] Connection header received
0806 16:08:33.695267 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:33.695270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:33.695316 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:33.695397 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:33.696557 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:33.697386 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:33.698709 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:33.699075 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:33.699175 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:33.702760 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:33.702766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:33.702772 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:33.702806 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:33.704903 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:33.705637 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:33.705643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:33.705647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:33.705767 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:33.706199 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:33.706204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:33.706206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:33.706429 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:33.706621 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:33.707461 (+ 840us) server_negotiation.cc:299] Negotiation successful
0806 16:08:33.707579 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 16:08:33.712680 12220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb5807d137fc42b289b763ba00da8e86: Generated new TSK 0
I20260806 16:08:33.713794 12220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:33.722494 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 12103
W20260806 16:08:33.736085 12083 connection.cc:535] client connection to 127.13.148.62:37839 recv error: Network error: failed to read from TLS socket (remote: 127.13.148.62:37839): Connection reset by peer (error 104)
2026-08-06T16:08:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1698 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:33 dist-test-slave-7kzw krb5kdc[12237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:33 dist-test-slave-7kzw krb5kdc[12237](info): set up 2 sockets
Aug 06 16:08:33 dist-test-slave-7kzw krb5kdc[12237](info): commencing operation
krb5kdc: starting...
W20260806 16:08:34.935204 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:35 dist-test-slave-7kzw krb5kdc[12237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:35.081653 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46501
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:39623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:35.196768 12253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:35.197105 12253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:35.197183 12253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:35.201171 12253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:35.201282 12253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:35.201323 12253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:35.201361 12253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:35.201403 12253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:35.206936 12253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:39623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:35.208122 12253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:35.209213 12253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:35.218641 12264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:35.218822 12253 server_base.cc:1034] running on GCE node
W20260806 16:08:35.218645 12262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:35.218667 12260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:35.219537 12253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:35.220634 12253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:35.221808 12253 hybrid_clock.cc:648] HybridClock initialized: now 1786032515221780 us; error 39 us; skew 500 ppm
Aug 06 16:08:35 dist-test-slave-7kzw krb5kdc[12237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032515, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:35.224932 12253 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:35.225958 12253 webserver.cc:460] Webserver started at http://127.13.148.62:35027/ using document root <none> and password file <none>
I20260806 16:08:35.226684 12253 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:35.226796 12253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:35.227095 12253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:35.229449 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "b6ff0e8506e3422398be8024d38644f2"
format_stamp: "Formatted at 2026-08-06 16:08:35 on dist-test-slave-7kzw"
server_key: "d698d1b9d2460d623674ad25cfd44c07"
server_key_iv: "9238975007d5bd8870ce88e6f0b61768"
server_key_version: "encryptionkey@0"
I20260806 16:08:35.230126 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "b6ff0e8506e3422398be8024d38644f2"
format_stamp: "Formatted at 2026-08-06 16:08:35 on dist-test-slave-7kzw"
server_key: "d698d1b9d2460d623674ad25cfd44c07"
server_key_iv: "9238975007d5bd8870ce88e6f0b61768"
server_key_version: "encryptionkey@0"
I20260806 16:08:35.234418 12253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 16:08:35.237097 12272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:35.238608 12253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 16:08:35.238766 12253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "b6ff0e8506e3422398be8024d38644f2"
format_stamp: "Formatted at 2026-08-06 16:08:35 on dist-test-slave-7kzw"
server_key: "d698d1b9d2460d623674ad25cfd44c07"
server_key_iv: "9238975007d5bd8870ce88e6f0b61768"
server_key_version: "encryptionkey@0"
I20260806 16:08:35.238907 12253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:35.256861 12253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:35.260372 12253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:35.260599 12253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:35.268338 12253 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46501
I20260806 16:08:35.268365 12348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46501 every 8 connection(s)
I20260806 16:08:35.269375 12253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:35.269750 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 12253
I20260806 16:08:35.269909 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:35.270198 13904 external_mini_cluster.cc:1428] Setting key fcb2fb93f86c27481c5e870fe5fe662d
I20260806 16:08:35.272533 12349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:35.278995 12349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2: Bootstrap starting.
I20260806 16:08:35.279683 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:35.271682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40719 (local address 127.13.148.62:46501)
0806 16:08:35.272110 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:35.272126 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:35.272164 (+ 38us) server_negotiation.cc:407] Connection header received
0806 16:08:35.272850 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:35.272869 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:35.273210 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:35.273485 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:35.273817 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:35.274873 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:35.276104 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:35.276611 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:35.276791 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:35.277752 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:35.277757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:35.277770 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:35.278182 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:35.278242 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:35.278267 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:35.278383 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:35.278639 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 16:08:35.278840 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":144,"threads_started":1}
I20260806 16:08:35.281212 12349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:35.282012 12349 log.cc:824] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:35.282953 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:35.284453 12349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2: No bootstrap required, opened a new log
I20260806 16:08:35.289064 12349 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ff0e8506e3422398be8024d38644f2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46501 } }
I20260806 16:08:35.289350 12349 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:35.289426 12349 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ff0e8506e3422398be8024d38644f2, State: Initialized, Role: FOLLOWER
I20260806 16:08:35.289806 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:35.285236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55771 (local address 127.13.148.62:46501)
0806 16:08:35.285374 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:35.285378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:35.285591 (+ 213us) server_negotiation.cc:407] Connection header received
0806 16:08:35.285714 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:35.285716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:35.285756 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:35.285820 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:35.286130 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:35.286765 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:35.287946 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:35.288176 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:35.288260 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:35.289163 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:35.289166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:35.289168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:35.289180 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:35.289205 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:35.289217 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:35.289293 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:35.289584 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 16:08:35.289652 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:08:35.289912 12349 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ff0e8506e3422398be8024d38644f2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46501 } }
I20260806 16:08:35.290086 12349 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:35.290155 12349 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:35.290258 12349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:35.291230 12349 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ff0e8506e3422398be8024d38644f2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46501 } }
I20260806 16:08:35.291579 12349 leader_election.cc:302] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ff0e8506e3422398be8024d38644f2; no voters:
I20260806 16:08:35.291947 12349 leader_election.cc:288] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:35.292052 12362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:35.292276 12362 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [term 1 LEADER]: Becoming Leader. State: Replica: b6ff0e8506e3422398be8024d38644f2, State: Running, Role: LEADER
I20260806 16:08:35.292583 12362 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ff0e8506e3422398be8024d38644f2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46501 } }
I20260806 16:08:35.293207 12349 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:35.294178 12365 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ff0e8506e3422398be8024d38644f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ff0e8506e3422398be8024d38644f2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46501 } } }
I20260806 16:08:35.294346 12365 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:35.294656 12366 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ff0e8506e3422398be8024d38644f2. Latest consensus state: current_term: 1 leader_uuid: "b6ff0e8506e3422398be8024d38644f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ff0e8506e3422398be8024d38644f2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46501 } } }
I20260806 16:08:35.294744 12366 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:35.294734 12375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:35.298393 12375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:35.303821 12375 catalog_manager.cc:1347] Generated new cluster ID: 6b6a8d95f3614b2cad6d0ad56210ca4c
I20260806 16:08:35.303891 12375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:35.310387 12375 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:35.311689 12375 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:35.321484 12375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ff0e8506e3422398be8024d38644f2: Generated new TSK 0
I20260806 16:08:35.322356 12375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:35.329497 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 12253
W20260806 16:08:35.341109 12235 connection.cc:535] client connection to 127.13.148.62:46501 recv error: Network error: failed to read from TLS socket (remote: 127.13.148.62:46501): Connection reset by peer (error 104)
2026-08-06T16:08:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1604 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:35 dist-test-slave-7kzw krb5kdc[12391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:35 dist-test-slave-7kzw krb5kdc[12391](info): set up 2 sockets
Aug 06 16:08:35 dist-test-slave-7kzw krb5kdc[12391](info): commencing operation
krb5kdc: starting...
W20260806 16:08:36.579464 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:36 dist-test-slave-7kzw krb5kdc[12391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:36.718928 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46033
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:36.838737 12408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:36.839039 12408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:36.839115 12408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:36.843057 12408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:36.843156 12408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:36.843207 12408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:36.843266 12408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:36.843307 12408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:36.848634 12408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:36.849702 12408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:36.850764 12408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:36.859419 12417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:36.859419 12420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:36.859439 12416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:36.860224 12408 server_base.cc:1034] running on GCE node
I20260806 16:08:36.860852 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:36.862010 12408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:36.863212 12408 hybrid_clock.cc:648] HybridClock initialized: now 1786032516863191 us; error 31 us; skew 500 ppm
Aug 06 16:08:36 dist-test-slave-7kzw krb5kdc[12391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032516, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:36.866163 12408 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:36.867125 12408 webserver.cc:460] Webserver started at http://127.13.148.62:45049/ using document root <none> and password file <none>
I20260806 16:08:36.867700 12408 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:36.867769 12408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:36.867949 12408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:36.870041 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "781dffc3edb946ac80983637f5df8627"
format_stamp: "Formatted at 2026-08-06 16:08:36 on dist-test-slave-7kzw"
server_key: "548ddcf66b7344d3ee96cc6aa9145982"
server_key_iv: "33f3a07fd1deb1e38300d2eeb96f0079"
server_key_version: "encryptionkey@0"
I20260806 16:08:36.870646 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "781dffc3edb946ac80983637f5df8627"
format_stamp: "Formatted at 2026-08-06 16:08:36 on dist-test-slave-7kzw"
server_key: "548ddcf66b7344d3ee96cc6aa9145982"
server_key_iv: "33f3a07fd1deb1e38300d2eeb96f0079"
server_key_version: "encryptionkey@0"
I20260806 16:08:36.874630 12408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 16:08:36.877835 12429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:36.879315 12408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 16:08:36.879441 12408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "781dffc3edb946ac80983637f5df8627"
format_stamp: "Formatted at 2026-08-06 16:08:36 on dist-test-slave-7kzw"
server_key: "548ddcf66b7344d3ee96cc6aa9145982"
server_key_iv: "33f3a07fd1deb1e38300d2eeb96f0079"
server_key_version: "encryptionkey@0"
I20260806 16:08:36.879545 12408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:36.885512 12408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:36.889232 12408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:36.889434 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:36.897953 12408 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46033
I20260806 16:08:36.897984 12494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46033 every 8 connection(s)
I20260806 16:08:36.899158 12408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:36.902280 12495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:36.905668 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 12408
I20260806 16:08:36.905827 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:36.906132 13904 external_mini_cluster.cc:1428] Setting key 7ea7f6dc41596ef9c4bce640833e73a8
I20260806 16:08:36.908968 12495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627: Bootstrap starting.
I20260806 16:08:36.911684 12495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:36.912076 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:36.907810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43761 (local address 127.13.148.62:46033)
0806 16:08:36.908355 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:36.908371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:36.908411 (+ 40us) server_negotiation.cc:407] Connection header received
0806 16:08:36.909216 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:36.909237 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:36.909537 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:36.909901 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:36.910795 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:36.910800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:36.910812 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:36.910823 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:36.910882 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:36.910902 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:36.911029 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:36.911136 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 16:08:36.911307 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":159,"threads_started":1}
I20260806 16:08:36.912537 12495 log.cc:824] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:36.914997 12495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627: No bootstrap required, opened a new log
I20260806 16:08:36.915315 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:36.919253 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:36.917411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53933 (local address 127.13.148.62:46033)
0806 16:08:36.917542 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:36.917547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:36.917793 (+ 246us) server_negotiation.cc:407] Connection header received
0806 16:08:36.917906 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:36.917908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:36.917937 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:36.918001 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:36.918848 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:36.918851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:36.918853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:36.918863 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:36.918887 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:36.918897 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:36.918967 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:36.919086 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 16:08:36.919117 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 16:08:36.919348 12495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781dffc3edb946ac80983637f5df8627" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46033 } }
I20260806 16:08:36.919674 12495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:36.919739 12495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781dffc3edb946ac80983637f5df8627, State: Initialized, Role: FOLLOWER
I20260806 16:08:36.920202 12495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781dffc3edb946ac80983637f5df8627" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46033 } }
I20260806 16:08:36.920367 12495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:36.920436 12495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:36.920533 12495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:36.921494 12495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781dffc3edb946ac80983637f5df8627" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46033 } }
I20260806 16:08:36.921844 12495 leader_election.cc:302] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 781dffc3edb946ac80983637f5df8627; no voters:
I20260806 16:08:36.922188 12495 leader_election.cc:288] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:36.922293 12506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:36.922557 12506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [term 1 LEADER]: Becoming Leader. State: Replica: 781dffc3edb946ac80983637f5df8627, State: Running, Role: LEADER
I20260806 16:08:36.922910 12506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781dffc3edb946ac80983637f5df8627" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46033 } }
I20260806 16:08:36.923529 12495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:36.924669 12509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781dffc3edb946ac80983637f5df8627. Latest consensus state: current_term: 1 leader_uuid: "781dffc3edb946ac80983637f5df8627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781dffc3edb946ac80983637f5df8627" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46033 } } }
I20260806 16:08:36.924804 12509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:36.925087 12507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "781dffc3edb946ac80983637f5df8627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781dffc3edb946ac80983637f5df8627" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46033 } } }
I20260806 16:08:36.925195 12507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:36.925500 12516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:36.928248 12516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:36.934011 12516 catalog_manager.cc:1347] Generated new cluster ID: 063918b7ac03493785d67e0390feebe5
I20260806 16:08:36.934128 12516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:36.948254 12516 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:36.948593 12516 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:36.957429 12516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 781dffc3edb946ac80983637f5df8627: Generated new TSK 0
I20260806 16:08:36.958248 12516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:36.963032 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 12408
2026-08-06T16:08:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1632 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:37 dist-test-slave-7kzw krb5kdc[12535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:37 dist-test-slave-7kzw krb5kdc[12535](info): set up 2 sockets
Aug 06 16:08:37 dist-test-slave-7kzw krb5kdc[12535](info): commencing operation
krb5kdc: starting...
W20260806 16:08:38.175084 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:38 dist-test-slave-7kzw krb5kdc[12535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:38.309988 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46415
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:38.423731 12557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:38.423991 12557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:38.424033 12557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:38.428015 12557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:38.428072 12557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:38.428092 12557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:38.428109 12557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:38.428164 12557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:38.433467 12557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:38.434571 12557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:38.435634 12557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:38.444384 12563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:38.444420 12566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:38.444571 12564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:38.444701 12557 server_base.cc:1034] running on GCE node
I20260806 16:08:38.445312 12557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:38.446400 12557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:38.447625 12557 hybrid_clock.cc:648] HybridClock initialized: now 1786032518447603 us; error 37 us; skew 500 ppm
Aug 06 16:08:38 dist-test-slave-7kzw krb5kdc[12535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032518, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:38.450436 12557 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:38.451463 12557 webserver.cc:460] Webserver started at http://127.13.148.62:33185/ using document root <none> and password file <none>
I20260806 16:08:38.452044 12557 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:38.452149 12557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:38.452389 12557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:38.454437 12557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a"
format_stamp: "Formatted at 2026-08-06 16:08:38 on dist-test-slave-7kzw"
server_key: "6dfd7e98ccaeca1c8afc75bc29a56b23"
server_key_iv: "359a9a001eedd890d5e19025aad4e2d9"
server_key_version: "encryptionkey@0"
I20260806 16:08:38.455089 12557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a"
format_stamp: "Formatted at 2026-08-06 16:08:38 on dist-test-slave-7kzw"
server_key: "6dfd7e98ccaeca1c8afc75bc29a56b23"
server_key_iv: "359a9a001eedd890d5e19025aad4e2d9"
server_key_version: "encryptionkey@0"
I20260806 16:08:38.459307 12557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:38.461747 12577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:38.462966 12557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:38.463110 12557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a"
format_stamp: "Formatted at 2026-08-06 16:08:38 on dist-test-slave-7kzw"
server_key: "6dfd7e98ccaeca1c8afc75bc29a56b23"
server_key_iv: "359a9a001eedd890d5e19025aad4e2d9"
server_key_version: "encryptionkey@0"
I20260806 16:08:38.463225 12557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:38.474936 12557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:38.478185 12557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:38.478406 12557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:38.485707 12557 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46415
I20260806 16:08:38.485728 12654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46415 every 8 connection(s)
I20260806 16:08:38.486711 12557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:38.487577 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 12557
I20260806 16:08:38.487739 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:38.488029 13904 external_mini_cluster.cc:1428] Setting key 47d754b2e684e036a0d65f96038f4109
I20260806 16:08:38.490234 12656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:38.497164 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a: Bootstrap starting.
Aug 06 16:08:38 dist-test-slave-7kzw krb5kdc[12535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:38.499962 12656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:38.500921 12656 log.cc:824] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:38.503547 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a: No bootstrap required, opened a new log
I20260806 16:08:38.504281 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:38.489429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46601 (local address 127.13.148.62:46415)
0806 16:08:38.489887 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:38.489909 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:38.490010 (+ 101us) server_negotiation.cc:407] Connection header received
0806 16:08:38.490945 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:38.490968 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:38.491405 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:38.491763 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:38.492479 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:38.493997 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:38.494971 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:38.495689 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:38.495852 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:38.498520 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:38.498527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:38.498544 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:38.498577 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:38.500687 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:38.501238 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:38.501246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:38.501252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:38.501408 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:38.501823 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:38.501825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:38.501826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:38.502393 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:38.502613 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:38.502949 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 16:08:38.503270 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":108,"threads_started":1}
I20260806 16:08:38.507520 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:38.508071 12656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46415 } }
I20260806 16:08:38.508312 12656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:38.508386 12656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5ca6aa0d26417b9b0dc66de5532a0a, State: Initialized, Role: FOLLOWER
I20260806 16:08:38.508865 12656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46415 } }
I20260806 16:08:38.509050 12656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:38.509156 12656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:38.509281 12656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:38.510622 12656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46415 } }
I20260806 16:08:38.511055 12656 leader_election.cc:302] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d5ca6aa0d26417b9b0dc66de5532a0a; no voters:
I20260806 16:08:38.511452 12656 leader_election.cc:288] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:38.511564 12668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:38.511801 12668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [term 1 LEADER]: Becoming Leader. State: Replica: 0d5ca6aa0d26417b9b0dc66de5532a0a, State: Running, Role: LEADER
I20260806 16:08:38.512180 12668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46415 } }
I20260806 16:08:38.512833 12656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:38.515027 12671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d5ca6aa0d26417b9b0dc66de5532a0a. Latest consensus state: current_term: 1 leader_uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46415 } } }
I20260806 16:08:38.515168 12671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:38.516535 12670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5ca6aa0d26417b9b0dc66de5532a0a" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46415 } } }
I20260806 16:08:38.516672 12670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a [sys.catalog]: This master's current role is: LEADER
W20260806 16:08:38.518601 12686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:38.518689 12686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:38.519047 12687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:38.520044 12687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:38.520640 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:38.509885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46323 (local address 127.13.148.62:46415)
0806 16:08:38.510018 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:38.510023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:38.510303 (+ 280us) server_negotiation.cc:407] Connection header received
0806 16:08:38.510423 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:38.510425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:38.510507 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:38.510588 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:38.511485 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:38.512292 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:38.513403 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:38.513798 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:38.513908 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:38.516450 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:38.516453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:38.516455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:38.516476 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:38.518254 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:38.518909 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:38.518913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:38.518915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:38.519008 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:38.519348 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:38.519352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:38.519353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:38.519579 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:38.519708 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:38.520368 (+ 660us) server_negotiation.cc:299] Negotiation successful
0806 16:08:38.520493 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:08:38.526121 12687 catalog_manager.cc:1347] Generated new cluster ID: c2205b33fa96499780fd1279e9419079
I20260806 16:08:38.526235 12687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:38.547465 12687 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:38.548483 12687 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:38.556232 12687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d5ca6aa0d26417b9b0dc66de5532a0a: Generated new TSK 0
I20260806 16:08:38.556974 12687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:38.565668 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 12557
2026-08-06T16:08:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1602 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:38 dist-test-slave-7kzw krb5kdc[12693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:38 dist-test-slave-7kzw krb5kdc[12693](info): set up 2 sockets
Aug 06 16:08:38 dist-test-slave-7kzw krb5kdc[12693](info): commencing operation
krb5kdc: starting...
W20260806 16:08:39.775228 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:39 dist-test-slave-7kzw krb5kdc[12693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:39.918076 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40173
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:40.036501 12710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:40.036988 12710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:40.037036 12710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:40.041488 12710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:40.041581 12710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:40.041608 12710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:40.041626 12710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:40.041642 12710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:40.047153 12710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:40.048269 12710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:40.049274 12710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:40.058697 12718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:40.058775 12715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:40.058775 12716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:40.059077 12710 server_base.cc:1034] running on GCE node
I20260806 16:08:40.059965 12710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:40.061368 12710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:40.062595 12710 hybrid_clock.cc:648] HybridClock initialized: now 1786032520062546 us; error 35 us; skew 500 ppm
Aug 06 16:08:40 dist-test-slave-7kzw krb5kdc[12693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032520, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:40.068871 12710 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:40.069792 12710 webserver.cc:460] Webserver started at http://127.13.148.62:33889/ using document root <none> and password file <none>
I20260806 16:08:40.070339 12710 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:40.070399 12710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:40.070657 12710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:40.072744 12710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "a8b71de405544e5e9df341eb5e517c88"
format_stamp: "Formatted at 2026-08-06 16:08:40 on dist-test-slave-7kzw"
server_key: "64969583adf5ba126b201aae5ad306f7"
server_key_iv: "c293244d9052cd70157c27d72829835c"
server_key_version: "encryptionkey@0"
I20260806 16:08:40.073273 12710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "a8b71de405544e5e9df341eb5e517c88"
format_stamp: "Formatted at 2026-08-06 16:08:40 on dist-test-slave-7kzw"
server_key: "64969583adf5ba126b201aae5ad306f7"
server_key_iv: "c293244d9052cd70157c27d72829835c"
server_key_version: "encryptionkey@0"
I20260806 16:08:40.077131 12710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 16:08:40.079826 12728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:40.081264 12710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:08:40.081403 12710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "a8b71de405544e5e9df341eb5e517c88"
format_stamp: "Formatted at 2026-08-06 16:08:40 on dist-test-slave-7kzw"
server_key: "64969583adf5ba126b201aae5ad306f7"
server_key_iv: "c293244d9052cd70157c27d72829835c"
server_key_version: "encryptionkey@0"
I20260806 16:08:40.081511 12710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:40.101115 12710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:40.104559 12710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:40.104751 12710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:40.112363 12710 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40173
I20260806 16:08:40.112401 12795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40173 every 8 connection(s)
I20260806 16:08:40.113415 12710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:40.116559 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 12710
I20260806 16:08:40.116617 12798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:40.116729 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:40.117049 13904 external_mini_cluster.cc:1428] Setting key 4ebcbfa987df9038410a308470f92cdd
I20260806 16:08:40.123739 12798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88: Bootstrap starting.
I20260806 16:08:40.126438 12798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:40.127229 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:40.118546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49061 (local address 127.13.148.62:40173)
0806 16:08:40.119027 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:40.119043 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:40.119149 (+ 106us) server_negotiation.cc:407] Connection header received
0806 16:08:40.120018 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:40.120038 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:40.120395 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:40.120675 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:40.121075 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:40.122077 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:40.123335 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:40.124089 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:40.124280 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:40.125173 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:40.125178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:40.125192 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:40.125644 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:40.125712 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:40.125738 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:40.125832 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:40.126007 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 16:08:40.126309 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":154,"threads_started":1}
I20260806 16:08:40.127249 12798 log.cc:824] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:40.129521 12798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88: No bootstrap required, opened a new log
I20260806 16:08:40.130275 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:40.133499 12798 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b71de405544e5e9df341eb5e517c88" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40173 } }
I20260806 16:08:40.133775 12798 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:40.133862 12798 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b71de405544e5e9df341eb5e517c88, State: Initialized, Role: FOLLOWER
I20260806 16:08:40.134303 12798 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b71de405544e5e9df341eb5e517c88" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40173 } }
I20260806 16:08:40.134467 12798 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:40.134579 12798 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:40.134706 12798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:40.135699 12798 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b71de405544e5e9df341eb5e517c88" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40173 } }
I20260806 16:08:40.136080 12798 leader_election.cc:302] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8b71de405544e5e9df341eb5e517c88; no voters:
I20260806 16:08:40.136453 12798 leader_election.cc:288] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:40.136565 12812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:40.136858 12812 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [term 1 LEADER]: Becoming Leader. State: Replica: a8b71de405544e5e9df341eb5e517c88, State: Running, Role: LEADER
I20260806 16:08:40.137331 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:40.132601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45155 (local address 127.13.148.62:40173)
0806 16:08:40.132769 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:40.132774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:40.132888 (+ 114us) server_negotiation.cc:407] Connection header received
0806 16:08:40.133099 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:40.133102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:40.133154 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:40.133258 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:40.133573 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:40.134313 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:40.135360 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:40.135636 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:40.135769 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:40.136789 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:40.136794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:40.136798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:40.136819 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:40.136854 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:40.136872 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:40.136963 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:40.137145 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 16:08:40.137203 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 16:08:40.137300 12812 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b71de405544e5e9df341eb5e517c88" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40173 } }
I20260806 16:08:40.137877 12798 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:40.138665 12812 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8b71de405544e5e9df341eb5e517c88. Latest consensus state: current_term: 1 leader_uuid: "a8b71de405544e5e9df341eb5e517c88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b71de405544e5e9df341eb5e517c88" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40173 } } }
I20260806 16:08:40.139508 12812 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:40.139374 12814 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8b71de405544e5e9df341eb5e517c88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b71de405544e5e9df341eb5e517c88" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40173 } } }
I20260806 16:08:40.139833 12814 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88 [sys.catalog]: This master's current role is: LEADER
W20260806 16:08:40.142403 12833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:40.142555 12833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:40.142640 12830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:40.143661 12830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:40.149679 12830 catalog_manager.cc:1347] Generated new cluster ID: bb3249bf374e45f38aec2c5b06474ddc
I20260806 16:08:40.149763 12830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:40.171191 12830 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:40.172366 12830 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:40.184489 12830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8b71de405544e5e9df341eb5e517c88: Generated new TSK 0
I20260806 16:08:40.185287 12830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:40.196622 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 12710
2026-08-06T16:08:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1630 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:40 dist-test-slave-7kzw krb5kdc[12842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:40 dist-test-slave-7kzw krb5kdc[12842](info): set up 2 sockets
Aug 06 16:08:40 dist-test-slave-7kzw krb5kdc[12842](info): commencing operation
krb5kdc: starting...
W20260806 16:08:41.482577 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:41 dist-test-slave-7kzw krb5kdc[12842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:41.622146 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42185
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:42185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:41.743808 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:41.744112 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:41.744181 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:41.748842 12858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:41.748960 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:41.749004 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:41.749064 12858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:41.749110 12858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:41.754860 12858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:42185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:41.755939 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:41.757050 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:41.765748 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:41.765751 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:41.765750 12869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:41.766078 12858 server_base.cc:1034] running on GCE node
I20260806 16:08:41.766762 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:41.767897 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:41.769093 12858 hybrid_clock.cc:648] HybridClock initialized: now 1786032521769077 us; error 30 us; skew 500 ppm
Aug 06 16:08:41 dist-test-slave-7kzw krb5kdc[12842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032521, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:41.771942 12858 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:41.772866 12858 webserver.cc:460] Webserver started at http://127.13.148.62:46039/ using document root <none> and password file <none>
I20260806 16:08:41.773398 12858 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:41.773461 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:41.773629 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:41.775668 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "37b5855aba28416f998988b836294b9d"
format_stamp: "Formatted at 2026-08-06 16:08:41 on dist-test-slave-7kzw"
server_key: "a02eb63c96822daeab764a42974571e3"
server_key_iv: "1fab5657568bc8301f8af978cd0e84b5"
server_key_version: "encryptionkey@0"
I20260806 16:08:41.776198 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "37b5855aba28416f998988b836294b9d"
format_stamp: "Formatted at 2026-08-06 16:08:41 on dist-test-slave-7kzw"
server_key: "a02eb63c96822daeab764a42974571e3"
server_key_iv: "1fab5657568bc8301f8af978cd0e84b5"
server_key_version: "encryptionkey@0"
I20260806 16:08:41.779860 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 16:08:41.782231 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:41.783555 12858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:41.783676 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "37b5855aba28416f998988b836294b9d"
format_stamp: "Formatted at 2026-08-06 16:08:41 on dist-test-slave-7kzw"
server_key: "a02eb63c96822daeab764a42974571e3"
server_key_iv: "1fab5657568bc8301f8af978cd0e84b5"
server_key_version: "encryptionkey@0"
I20260806 16:08:41.783762 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:41.788379 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:41.791463 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:41.791615 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:41.799281 12858 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:42185
I20260806 16:08:41.799294 12946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:42185 every 8 connection(s)
I20260806 16:08:41.800264 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:41.803267 12947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:41.809240 12947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d: Bootstrap starting.
I20260806 16:08:41.810606 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 12858
I20260806 16:08:41.810758 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:41.811045 13904 external_mini_cluster.cc:1428] Setting key 8a049c16bca80784815c6068bd6f5bc9
I20260806 16:08:41.812357 12947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:41.813196 12947 log.cc:824] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:41.816010 12947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d: No bootstrap required, opened a new log
I20260806 16:08:41.816911 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:41.812298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41133 (local address 127.13.148.62:42185)
0806 16:08:41.812809 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:41.812827 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:41.812867 (+ 40us) server_negotiation.cc:407] Connection header received
0806 16:08:41.813720 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:41.813749 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:41.814146 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:41.814658 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:41.815616 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:41.815622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:41.815642 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:41.815655 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:41.815712 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:41.815733 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:41.815861 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:41.815976 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 16:08:41.816147 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":157,"threads_started":1}
I20260806 16:08:41.819880 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:41.819900 12947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5855aba28416f998988b836294b9d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42185 } }
I20260806 16:08:41.820158 12947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 16:08:41.820248 12947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b5855aba28416f998988b836294b9d, State: Initialized, Role: FOLLOWER
I20260806 16:08:41.820679 12947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5855aba28416f998988b836294b9d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42185 } }
I20260806 16:08:41.820809 12947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 16:08:41.820894 12947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 16:08:41.821027 12947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:41.822079 12947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5855aba28416f998988b836294b9d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42185 } }
I20260806 16:08:41.822455 12947 leader_election.cc:302] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37b5855aba28416f998988b836294b9d; no voters:
I20260806 16:08:41.822851 12947 leader_election.cc:288] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 16:08:41.822932 12961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:41.823205 12961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [term 1 LEADER]: Becoming Leader. State: Replica: 37b5855aba28416f998988b836294b9d, State: Running, Role: LEADER
I20260806 16:08:41.823566 12961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5855aba28416f998988b836294b9d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42185 } }
I20260806 16:08:41.824409 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:41.822665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60043 (local address 127.13.148.62:42185)
0806 16:08:41.822797 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:41.822801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:41.822820 (+ 19us) server_negotiation.cc:407] Connection header received
0806 16:08:41.822890 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:41.822893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:41.822931 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:41.823020 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:41.823962 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:41.823966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:41.823969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 16:08:41.823981 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:41.824011 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 16:08:41.824025 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:41.824131 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:41.824248 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 16:08:41.824288 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 16:08:41.824779 12947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:41.825712 12962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37b5855aba28416f998988b836294b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5855aba28416f998988b836294b9d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42185 } } }
I20260806 16:08:41.825867 12962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:41.826306 12971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:41.826264 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37b5855aba28416f998988b836294b9d. Latest consensus state: current_term: 1 leader_uuid: "37b5855aba28416f998988b836294b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b5855aba28416f998988b836294b9d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42185 } } }
I20260806 16:08:41.826520 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:41.829830 12971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:41.836033 12971 catalog_manager.cc:1347] Generated new cluster ID: 365e7b79579345248c3a8d1e5214b542
I20260806 16:08:41.836120 12971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:41.846936 12971 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:41.847263 12971 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:41.856657 12971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37b5855aba28416f998988b836294b9d: Generated new TSK 0
I20260806 16:08:41.857429 12971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 16:08:41.861817 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 12858
2026-08-06T16:08:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1666 ms)
[----------] 18 tests from AuthTokenIssuingTest (29534 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:41 dist-test-slave-7kzw krb5kdc[12987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:41 dist-test-slave-7kzw krb5kdc[12987](info): set up 2 sockets
Aug 06 16:08:41 dist-test-slave-7kzw krb5kdc[12987](info): commencing operation
krb5kdc: starting...
W20260806 16:08:43.117240 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:43.258209 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:43711
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:43711,127.13.148.61:34453,127.13.148.60:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:43.374874 13008 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:08:43.375190 13008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:43.375216 13008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:43.375255 13008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:43.379289 13008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:43.379359 13008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:43.379379 13008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:43.379396 13008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:43.379412 13008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:43.384814 13008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:43711,127.13.148.61:34453,127.13.148.60:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:43.386006 13008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:43.387135 13008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:43.396088 13013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:43.396184 13017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:43.396178 13014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:43.396449 13008 server_base.cc:1034] running on GCE node
I20260806 16:08:43.397058 13008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:43.398209 13008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:43.399448 13008 hybrid_clock.cc:648] HybridClock initialized: now 1786032523399417 us; error 49 us; skew 500 ppm
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:43.402529 13008 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:43.403467 13008 webserver.cc:460] Webserver started at http://127.13.148.62:33181/ using document root <none> and password file <none>
I20260806 16:08:43.404085 13008 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:43.404184 13008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:43.404433 13008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:43.406723 13008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "014bd1a47b8043668331e000663f6810"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "521d372b7ef95760bb2a48092931d277"
server_key_iv: "2a0613e5f914343dd5f9eefee1f4fdfd"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.407372 13008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "014bd1a47b8043668331e000663f6810"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "521d372b7ef95760bb2a48092931d277"
server_key_iv: "2a0613e5f914343dd5f9eefee1f4fdfd"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.411375 13008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:43.413964 13026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:43.415344 13008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 16:08:43.415496 13008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "014bd1a47b8043668331e000663f6810"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "521d372b7ef95760bb2a48092931d277"
server_key_iv: "2a0613e5f914343dd5f9eefee1f4fdfd"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.415623 13008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:43.435003 13008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:43.438568 13008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:43.438797 13008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:43.446318 13008 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:43711
I20260806 16:08:43.446661 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:43711 every 8 connection(s)
I20260806 16:08:43.447417 13008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:43.450310 13093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:43.454023 13093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:43.456725 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 13008
I20260806 16:08:43.456879 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:43.457198 13904 external_mini_cluster.cc:1428] Setting key 78371d0154d37d4a91006223031bf85d
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.61@KRBTEST.COM" created.
I20260806 16:08:43.470952 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.456279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43711 (local address 127.0.0.1:59398)
0806 16:08:43.456957 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.457023 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:43.458089 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:43.458912 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:43.458918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:43.459443 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:43.459718 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.459732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.461033 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.461036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.461837 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.461846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.462629 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.462634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.462814 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.463527 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:43.463545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:43.465163 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:43.467460 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.467472 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.467484 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.467738 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.468121 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.468124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.468126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.468258 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.468917 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:43.468924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:43.469097 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 16:08:43.469293 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 16:08:43.469702 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":177,"thread_start_us":180,"threads_started":1}
I20260806 16:08:43.470952 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.457369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.13.148.62:43711)
0806 16:08:43.457720 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.457734 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.457785 (+ 51us) server_negotiation.cc:407] Connection header received
0806 16:08:43.458447 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.458452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.458711 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.458913 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.459893 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.460858 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.461972 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.462394 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.462640 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.465322 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:43.465329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:43.465343 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:43.465375 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:43.467342 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.467890 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.467895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.467897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.468008 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.468370 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.468374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.468376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.468783 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:43.468932 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:43.469386 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 16:08:43.469642 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":225,"thread_start_us":120,"threads_started":1}
I20260806 16:08:43.473968 13093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:43.474932 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.474583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53386)
0806 16:08:43.474759 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.474855 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260806 16:08:43.475194 13027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
I20260806 16:08:43.476207 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.475967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53394)
0806 16:08:43.476087 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.476148 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 16:08:43.476363 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:34453: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
I20260806 16:08:43.491441 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:34453
--webserver_interface=127.13.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:43711,127.13.148.61:34453,127.13.148.60:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:08:43.525641 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } attempt: 1
I20260806 16:08:43.526674 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.526244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53404)
0806 16:08:43.526414 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.526544 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 16:08:43.527546 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.527305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53416)
0806 16:08:43.527425 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.527483 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 16:08:43.527752 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:34453: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
I20260806 16:08:43.595973 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } attempt: 2
I20260806 16:08:43.596925 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.596588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53422)
0806 16:08:43.596773 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.596838 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 16:08:43.597780 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.597505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53438)
0806 16:08:43.597655 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.597714 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 16:08:43.597960 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:34453: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
W20260806 16:08:43.609009 13101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:08:43.609288 13101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:43.609313 13101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:43.609349 13101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:43.613322 13101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:43.613384 13101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:43.613404 13101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:43.613420 13101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:43.613435 13101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:43.618866 13101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:43711,127.13.148.61:34453,127.13.148.60:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:34453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:43.619983 13101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:43.620987 13101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:43.629614 13110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:43.629652 13108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:43.629623 13107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:43.630088 13101 server_base.cc:1034] running on GCE node
I20260806 16:08:43.630769 13101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:43.631886 13101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:43.633059 13101 hybrid_clock.cc:648] HybridClock initialized: now 1786032523633032 us; error 39 us; skew 500 ppm
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:43.641723 13101 init.cc:377] Logged in from keytab as kudu/127.13.148.61@KRBTEST.COM (short username kudu)
I20260806 16:08:43.642920 13101 webserver.cc:460] Webserver started at http://127.13.148.61:41417/ using document root <none> and password file <none>
I20260806 16:08:43.643558 13101 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:43.643643 13101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:43.643890 13101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:43.646384 13101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/data/instance:
uuid: "df5294b228a247c995937c01fa4b8e10"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "7b73c25615273ac9732f8fea58c4069e"
server_key_iv: "e7af4d95519f0858f733d7ba32fa4fb1"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.647112 13101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/wal/instance:
uuid: "df5294b228a247c995937c01fa4b8e10"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "7b73c25615273ac9732f8fea58c4069e"
server_key_iv: "e7af4d95519f0858f733d7ba32fa4fb1"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.651494 13101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:43.654171 13118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:43.655656 13101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:08:43.655813 13101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/wal
uuid: "df5294b228a247c995937c01fa4b8e10"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "7b73c25615273ac9732f8fea58c4069e"
server_key_iv: "e7af4d95519f0858f733d7ba32fa4fb1"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.655943 13101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:43.689153 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } attempt: 3
I20260806 16:08:43.690197 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.689792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53450)
0806 16:08:43.689982 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.690066 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 16:08:43.691068 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.690825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53458)
0806 16:08:43.690943 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.690999 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 16:08:43.691246 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:34453: Network error: Client connection negotiation failed: client connection to 127.13.148.61:34453: connect: Connection refused (error 111)
I20260806 16:08:43.704418 13101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:43.708266 13101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:43.708501 13101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:43.716817 13101 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:34453
I20260806 16:08:43.717175 13194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:34453 every 8 connection(s)
I20260806 16:08:43.718019 13101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
I20260806 16:08:43.720206 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 13101
I20260806 16:08:43.720369 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-1/wal/instance
I20260806 16:08:43.720691 13904 external_mini_cluster.cc:1428] Setting key 5159e87c3f0d10e35905a5c072ee2cb4
I20260806 16:08:43.721302 13195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:43.727249 13195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.148.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.60@KRBTEST.COM" created.
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:43.750753 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.730075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.13.148.62:43711)
0806 16:08:43.730269 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.730277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.731270 (+ 993us) server_negotiation.cc:407] Connection header received
0806 16:08:43.732688 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.732692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.732784 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.732910 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.734867 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.735589 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.737501 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.738005 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.738130 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.745320 (+ 7190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:43.745324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:43.745327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:43.745361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:43.747380 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.748241 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.748245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.748247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.748349 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.748929 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.748935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.748938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.749206 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:43.749339 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:43.750294 (+ 955us) server_negotiation.cc:299] Negotiation successful
0806 16:08:43.750509 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 16:08:43.752724 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.730429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43711 (local address 127.0.0.1:59400)
0806 16:08:43.731082 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.731136 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:43.732393 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:43.733209 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:43.733221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:43.733766 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:43.734632 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.734662 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.735752 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.735755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.737348 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.737365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.738382 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.738386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.738681 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.739424 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:43.739445 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:43.745001 (+ 5556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:43.747538 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.747551 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.747576 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.748079 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.748468 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.748475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.748480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.748786 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.749375 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:43.749383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:43.749879 (+ 496us) client_negotiation.cc:769] Sending connection context
0806 16:08:43.750978 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0806 16:08:43.751438 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":143,"threads_started":1}
I20260806 16:08:43.754814 13195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
I20260806 16:08:43.758903 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:36513
--webserver_interface=127.13.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:43711,127.13.148.61:34453,127.13.148.60:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:08:43.768455 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.755471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53472)
0806 16:08:43.756011 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.756029 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:43.756128 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:43.756626 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:43.756629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:43.756837 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:43.756958 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.756965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.758629 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.758636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.759625 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.759642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.760824 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.760831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.760952 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.761530 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:43.761543 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:43.763141 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:43.766059 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.766067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.766070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.766374 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.766813 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.766820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.766825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.766929 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.767722 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:43.767728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:43.767851 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 16:08:43.767984 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 16:08:43.768168 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485}
I20260806 16:08:43.769251 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.755619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.13.148.61:34453)
0806 16:08:43.756012 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.756026 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.756226 (+ 200us) server_negotiation.cc:407] Connection header received
0806 16:08:43.756263 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.756269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.756499 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.756662 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.757088 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.758435 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.759788 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.760501 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.760632 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.763307 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:43.763322 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:43.763344 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:43.763378 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:43.765906 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.766554 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.766564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.766568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.766690 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.767043 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.767048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.767051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.767488 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:43.767671 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:43.768795 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0806 16:08:43.768982 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":171,"threads_started":1}
I20260806 16:08:43.774569 13195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:43.775514 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.775215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38246)
0806 16:08:43.775335 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.775449 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 16:08:43.775875 13120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
I20260806 16:08:43.776824 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.776599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38262)
0806 16:08:43.776705 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.776757 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 16:08:43.777064 13195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:36513: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
I20260806 16:08:43.826395 13195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } attempt: 1
I20260806 16:08:43.827340 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.827032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38264)
0806 16:08:43.827182 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.827247 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 16:08:43.828102 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.827885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38278)
0806 16:08:43.827993 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.828038 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 16:08:43.828287 13195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:36513: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
I20260806 16:08:43.834443 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } attempt: 4
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:08:43.845321 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.835031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53474)
0806 16:08:43.835218 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.835237 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:43.835367 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:43.835714 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:43.835717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:43.835912 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:43.836021 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.836027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.837550 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.837553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.838732 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.838747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.839368 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.839371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.839466 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.840093 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:43.840107 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:43.841667 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:43.843671 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.843674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.843676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.843884 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.844211 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.844215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.844216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.844316 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.844804 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:43.844810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:43.844934 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 16:08:43.845063 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 16:08:43.845191 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 16:08:43.845420 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.835105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.13.148.61:34453)
0806 16:08:43.835229 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.835234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.835343 (+ 109us) server_negotiation.cc:407] Connection header received
0806 16:08:43.835465 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.835469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.835521 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.835618 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.836158 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.837400 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.838869 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.839239 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.839354 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.841823 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:43.841827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:43.841830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:43.841862 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:43.843558 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.843984 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.843988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.843990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.844083 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.844422 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.844428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.844431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.844670 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:43.844773 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:43.845139 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 16:08:43.845266 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 16:08:43.846168 13093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:43.847004 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.846750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38286)
0806 16:08:43.846886 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.846942 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 16:08:43.847811 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.847535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38294)
0806 16:08:43.847691 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.847751 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260806 16:08:43.848019 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:36513: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
I20260806 16:08:43.896478 13195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } attempt: 2
I20260806 16:08:43.897365 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.897049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38308)
0806 16:08:43.897216 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.897284 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 16:08:43.897398 13203 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:08:43.897799 13203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:43.897876 13203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:43.897957 13203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:08:43.898285 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.898064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38318)
0806 16:08:43.898165 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.898232 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 16:08:43.898417 13195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:36513: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
W20260806 16:08:43.902158 13203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:43.902249 13203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:43.902289 13203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:43.902325 13203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:43.902370 13203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:43.907198 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } attempt: 1
I20260806 16:08:43.908038 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.907729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38324)
0806 16:08:43.907888 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.907961 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 16:08:43.908071 13203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:43711,127.13.148.61:34453,127.13.148.60:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:43.908927 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.908676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38332)
0806 16:08:43.908804 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.908864 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 16:08:43.909137 13093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:36513: Network error: Client connection negotiation failed: client connection to 127.13.148.60:36513: connect: Connection refused (error 111)
I20260806 16:08:43.909348 13203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:43.910346 13203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:08:43.919489 13203 server_base.cc:1034] running on GCE node
W20260806 16:08:43.919430 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:43.919480 13210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:43.919405 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:43.920542 13203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:43.921768 13203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:43.923017 13203 hybrid_clock.cc:648] HybridClock initialized: now 1786032523923001 us; error 32 us; skew 500 ppm
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:43.926167 13203 init.cc:377] Logged in from keytab as kudu/127.13.148.60@KRBTEST.COM (short username kudu)
I20260806 16:08:43.927208 13203 webserver.cc:460] Webserver started at http://127.13.148.60:40375/ using document root <none> and password file <none>
I20260806 16:08:43.927809 13203 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:43.927881 13203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:43.928138 13203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:43.930523 13203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/data/instance:
uuid: "8b4b917f14e1439684d7c60f0fc7b9bf"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "638ac05d88763e4f25943f5fd54e6dd8"
server_key_iv: "25030e5d83f59caa260be5e79f3ffa4b"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.931128 13203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/wal/instance:
uuid: "8b4b917f14e1439684d7c60f0fc7b9bf"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "638ac05d88763e4f25943f5fd54e6dd8"
server_key_iv: "25030e5d83f59caa260be5e79f3ffa4b"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.935204 13203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:43.937902 13222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:43.939417 13203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 16:08:43.939576 13203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/wal
uuid: "8b4b917f14e1439684d7c60f0fc7b9bf"
format_stamp: "Formatted at 2026-08-06 16:08:43 on dist-test-slave-7kzw"
server_key: "638ac05d88763e4f25943f5fd54e6dd8"
server_key_iv: "25030e5d83f59caa260be5e79f3ffa4b"
server_key_version: "encryptionkey@0"
I20260806 16:08:43.939709 13203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:43.957991 13203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:43.961506 13203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:43.961719 13203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:43.969296 13203 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:36513
I20260806 16:08:43.969640 13292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:36513 every 8 connection(s)
I20260806 16:08:43.970352 13203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
I20260806 16:08:43.973609 13293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:43.976332 13093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } attempt: 2
I20260806 16:08:43.978173 13293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:43.979131 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 13203
I20260806 16:08:43.979305 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786032227601881-13904-0/minicluster-data/master-2/wal/instance
I20260806 16:08:43.979629 13904 external_mini_cluster.cc:1428] Setting key 49a0ea77a25c14650fbe1575ff6447f2
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:08:43.989620 13195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } attempt: 3
I20260806 16:08:43.992766 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.977006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38346)
0806 16:08:43.977198 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.977221 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:43.977362 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:43.980186 (+ 2824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:43.980190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:43.980435 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:43.980556 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.980565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.982610 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.982615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.983545 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.983554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.984339 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.984344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.984431 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.985030 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:43.985047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:43.986942 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:43.989746 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.989751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.989755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.990092 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.990630 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.990636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.990640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.990755 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.992030 (+ 1275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:43.992035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:43.992161 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 16:08:43.992353 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 16:08:43.992515 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 16:08:43.993594 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.981199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.13.148.62:43711)
0806 16:08:43.981319 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.981324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.981477 (+ 153us) server_negotiation.cc:407] Connection header received
0806 16:08:43.981616 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.981620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.981673 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.981786 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.982549 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.983388 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.985014 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.985376 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.985489 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.988390 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:43.988396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:43.988399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:43.988426 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:43.990432 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.991133 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.991138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.991141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.991244 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.992173 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.992179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.992181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.992511 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:43.992608 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:43.993309 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 16:08:43.993439 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:08:43.994585 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.980305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43711 (local address 127.0.0.1:59414)
0806 16:08:43.981328 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.981382 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:43.981512 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:43.981847 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:43.981855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:43.982230 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:43.982380 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.982390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.983514 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.983517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.984551 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.984560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.985557 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.985566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.985712 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.986333 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:43.986350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:43.988201 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:43.990592 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.990602 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.990610 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.991009 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.991906 (+ 897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:43.991911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:43.991915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:43.992036 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:43.992705 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:43.992712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:43.993036 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 16:08:43.993798 (+ 762us) client_negotiation.cc:240] Negotiation successful
0806 16:08:43.994007 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":163,"threads_started":1}
I20260806 16:08:43.994591 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.977756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.13.148.60:36513)
0806 16:08:43.978246 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.978265 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.978304 (+ 39us) server_negotiation.cc:407] Connection header received
0806 16:08:43.979310 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.979340 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.979842 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.980258 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.980763 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.982499 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.983664 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.984202 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.984391 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.987122 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:43.987129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:43.987142 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:43.987173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:43.989214 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.990262 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.990272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.990280 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.990451 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.990896 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.990899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.990905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.991884 (+ 979us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:43.992987 (+ 1103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:43.993134 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 16:08:43.993446 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":323,"thread_start_us":174,"threads_started":1}
I20260806 16:08:43.995965 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.980735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.13.148.60:36513)
0806 16:08:43.981318 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.981328 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.981344 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:08:43.981388 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.981391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.981425 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.981545 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.982657 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.983195 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.984517 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.984835 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.984956 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.988957 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:43.988963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:43.988967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:43.988998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:43.992569 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.993876 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.993879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.993882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.994013 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:43.994369 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:43.994373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:43.994375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:43.994632 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:43.995237 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:43.995639 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 16:08:43.995768 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":80,"threads_started":1}
I20260806 16:08:43.996129 13293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:43.999562 13093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810: Bootstrap starting.
Aug 06 16:08:43 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:08:44.002918 13093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810: Neither blocks nor log segments found. Creating new log.
Aug 06 16:08:44 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:08:44.003991 13093 log.cc:824] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:44.004885 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.990199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38372)
0806 16:08:43.990545 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.990562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:43.990642 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:43.991392 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:43.991396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:43.991608 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:43.991747 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.991756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.993760 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.993764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.994674 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.994689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.997986 (+ 3297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.997990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.998087 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:43.998697 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:43.998713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:44.000294 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:44.002829 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:44.002840 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:44.002845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:44.003114 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:44.003507 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:44.003511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:44.003513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:44.003623 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:44.004093 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:44.004098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:44.004199 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 16:08:44.004286 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 16:08:44.004647 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260806 16:08:44.005098 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.990827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.13.148.60:36513)
0806 16:08:43.991158 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.991166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.991177 (+ 11us) server_negotiation.cc:407] Connection header received
0806 16:08:43.991204 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.991206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.991249 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.991362 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.992795 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.993591 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.997485 (+ 3894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.997846 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.998310 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.000449 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:44.000453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:44.000455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:44.000480 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:44.002674 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.003299 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.003303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.003306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.003397 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.003738 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.003741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.003743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.003963 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:44.004507 (+ 544us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:44.004619 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 16:08:44.004742 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
I20260806 16:08:44.006394 13093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810: No bootstrap required, opened a new log
Aug 06 16:08:44 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:44.008109 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.996799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.13.148.61:34453)
0806 16:08:43.996922 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.996927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.996939 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:08:43.996975 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.996978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.997018 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.997107 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:43.997610 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.998376 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.999631 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.999986 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.000074 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.003639 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:44.003643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:44.003646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:44.003672 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:44.006012 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.006603 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.006607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.006608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.006699 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.007064 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.007067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.007069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.007355 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:44.007449 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:44.007821 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 16:08:44.007953 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 16:08:44.008652 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.996654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34453 (local address 127.0.0.1:53486)
0806 16:08:43.996777 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:43.996797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:43.996871 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:43.997144 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:43.997149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:43.997351 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:43.997476 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.997483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:43.998549 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:43.998553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:43.999483 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:43.999496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.000267 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.000273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:44.000408 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.001730 (+ 1322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:44.001747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:44.003453 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:44.006143 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:44.006147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:44.006149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:44.006393 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:44.006826 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:44.006829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:44.006831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:44.006951 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:44.007512 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:44.007514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:44.007620 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 16:08:44.008330 (+ 710us) client_negotiation.cc:240] Negotiation successful
0806 16:08:44.008483 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 16:08:44.009302 13195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10: Bootstrap starting.
I20260806 16:08:44.010448 13093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } }
I20260806 16:08:44.010754 13093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014bd1a47b8043668331e000663f6810, State: Initialized, Role: FOLLOWER
I20260806 16:08:44.011098 13093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } }
I20260806 16:08:44.011618 13303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.011791 13303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:44.011902 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:43.998582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.13.148.62:43711)
0806 16:08:43.998697 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:43.998701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:43.999491 (+ 790us) server_negotiation.cc:407] Connection header received
0806 16:08:43.999698 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:43.999701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:43.999756 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:43.999807 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.000791 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.001521 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.003146 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.003526 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.003611 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.007584 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:44.007589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:44.007592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:44.007622 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:44.009580 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.010174 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.010178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.010181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.010280 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.010664 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.010668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.010670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.010959 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:44.011142 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:44.011552 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 16:08:44.011690 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 16:08:44.012543 13093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:44.012614 13195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:44.013633 13195 log.cc:824] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:44.014804 13293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:44.016677 13195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10: No bootstrap required, opened a new log
W20260806 16:08:44.018882 13322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:44.018962 13322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 06 16:08:44 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:08:44.022066 13195 raft_consensus.cc:348] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } }
I20260806 16:08:44.022365 13195 raft_consensus.cc:729] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5294b228a247c995937c01fa4b8e10, State: Initialized, Role: FOLLOWER
I20260806 16:08:44.018541 13008 master_runner.cc:186] Error getting master registration for 127.13.148.61:34453: OK, instance_id { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" instance_seqno: 1786032523714540 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:08:44.022890 13008 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:08:44.022910 13195 consensus_queue.cc:260] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } }
I20260806 16:08:44.024788 13195 sys_catalog.cc:564] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 16:08:44 dist-test-slave-7kzw krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:08:44.025163 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:44.015328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:36513 (local address 127.0.0.1:38380)
0806 16:08:44.015578 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:44.015592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:44.015670 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:44.016205 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:44.016208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:44.016441 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:44.016538 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:44.016545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.017626 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.017629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:44.018564 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:44.018576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.019222 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.019226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:44.019325 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.019911 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:44.019929 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:44.021542 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:44.023670 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:44.023673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:44.023675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:44.023906 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:44.024188 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:44.024190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:44.024192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:44.024273 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:44.024694 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:44.024697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:44.024777 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 16:08:44.024876 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 16:08:44.025007 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260806 16:08:44.025604 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:44.015818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38380 (local address 127.13.148.60:36513)
0806 16:08:44.015932 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.015937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.015951 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:08:44.015986 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.015989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.016029 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.016107 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.016672 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.017497 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.018704 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.019083 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.019205 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.021689 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:44.021692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:44.021694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:44.021717 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:44.023552 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.024007 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.024010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.024012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.024084 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.024361 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.024365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.024366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.024567 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:44.024661 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:44.025346 (+ 685us) server_negotiation.cc:299] Negotiation successful
0806 16:08:44.025461 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 16:08:44.028836 13293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf: Bootstrap starting.
I20260806 16:08:44.029932 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.030083 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:44.031337 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:44.014438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.13.148.61:34453)
0806 16:08:44.014685 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.014690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.014703 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:08:44.014741 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.014755 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.014803 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.014918 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.020058 (+ 5140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.020873 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.022067 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.022433 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.022562 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.026517 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:44.026521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:44.026525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:44.026560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:44.028625 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.029307 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.029314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.029317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.029427 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.029750 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.029755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.029756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.030001 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:44.030983 (+ 982us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:44.031066 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 16:08:44.031184 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260806 16:08:44.031826 13293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:44.032766 13293 log.cc:824] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf: Log is configured to *not* fsync() on all Append() calls
W20260806 16:08:44.033692 13338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:44.033789 13338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:44.035593 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:44.036916 13059 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "014bd1a47b8043668331e000663f6810"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59418
I20260806 16:08:44.037179 13059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 16:08:44.037295 13059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:44.033735 13101 master_runner.cc:186] Error getting master registration for 127.13.148.60:36513: OK, instance_id { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" instance_seqno: 1786032523967209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:08:44.037797 13101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:08:44.038708 13059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } }
I20260806 16:08:44.039417 13293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf: No bootstrap required, opened a new log
I20260806 16:08:44.040549 13156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "014bd1a47b8043668331e000663f6810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "df5294b228a247c995937c01fa4b8e10"
I20260806 16:08:44.040782 13156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:44.042096 13156 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014bd1a47b8043668331e000663f6810 in term 1.
I20260806 16:08:44.042780 13027 leader_election.cc:302] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014bd1a47b8043668331e000663f6810, df5294b228a247c995937c01fa4b8e10; no voters:
I20260806 16:08:44.043164 13303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:44.043761 13303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [term 1 LEADER]: Becoming Leader. State: Replica: 014bd1a47b8043668331e000663f6810, State: Running, Role: LEADER
I20260806 16:08:44.044152 13303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } }
I20260806 16:08:44.044665 13293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } }
I20260806 16:08:44.044909 13293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4b917f14e1439684d7c60f0fc7b9bf, State: Initialized, Role: FOLLOWER
I20260806 16:08:44.045225 13293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } }
I20260806 16:08:44.046075 13303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 014bd1a47b8043668331e000663f6810. Latest consensus state: current_term: 1 leader_uuid: "014bd1a47b8043668331e000663f6810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.046188 13303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:44.046708 13293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:44.046737 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "014bd1a47b8043668331e000663f6810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf"
I20260806 16:08:44.046945 13253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:44.048393 13253 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014bd1a47b8043668331e000663f6810 in term 1.
I20260806 16:08:44.049060 13059 leader_election.cc:288] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [CANDIDATE]: Term 1 election: Requested vote from peers df5294b228a247c995937c01fa4b8e10 (127.13.148.61:34453), 8b4b917f14e1439684d7c60f0fc7b9bf (127.13.148.60:36513)
I20260806 16:08:44.054724 13348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:44.055794 13348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:44.060825 13156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [term 1 FOLLOWER]: Refusing update from remote peer 014bd1a47b8043668331e000663f6810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:44.060825 13253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [term 1 FOLLOWER]: Refusing update from remote peer 014bd1a47b8043668331e000663f6810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:44.061506 13303 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:44.061887 13303 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:44.066545 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 014bd1a47b8043668331e000663f6810. Latest consensus state: current_term: 1 leader_uuid: "014bd1a47b8043668331e000663f6810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.066696 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:44.067543 13378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "014bd1a47b8043668331e000663f6810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.067695 13378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:44.067804 13378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 014bd1a47b8043668331e000663f6810. Latest consensus state: current_term: 1 leader_uuid: "014bd1a47b8043668331e000663f6810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.067898 13378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:44.070964 13303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "014bd1a47b8043668331e000663f6810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.071115 13303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:44.072780 13378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "014bd1a47b8043668331e000663f6810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.072901 13378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:44.073052 13377 mvcc.cc:205] Tried to move back new op lower bound from 7315589218546098176 to 7315589218488877056. Current Snapshot: MvccSnapshot[applied={T|T < 7315589218546098176}]
I20260806 16:08:44.073590 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:44.052146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.13.148.61:34453)
0806 16:08:44.052265 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.052269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.058375 (+ 6106us) server_negotiation.cc:407] Connection header received
0806 16:08:44.058602 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.058605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.058656 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.058707 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.065371 (+ 6664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.066077 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.067256 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.067602 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.067685 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.069679 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:44.069684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:44.069687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:44.069715 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:44.071517 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.072041 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.072045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.072047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.072146 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.072496 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.072499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.072501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.072745 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:44.072866 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:44.073250 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 16:08:44.073380 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 16:08:44.074852 13379 mvcc.cc:205] Tried to move back new op lower bound from 7315589218546098176 to 7315589218488877056. Current Snapshot: MvccSnapshot[applied={T|T < 7315589218546098176}]
W20260806 16:08:44.077944 13365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 16:08:44.078121 13348 catalog_manager.cc:1347] Generated new cluster ID: 62407cfb29274de7a68ea6c827e35cab
I20260806 16:08:44.078182 13348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:44.081655 13303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "014bd1a47b8043668331e000663f6810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.081780 13303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:44.082861 13383 mvcc.cc:205] Tried to move back new op lower bound from 7315589218546098176 to 7315589218488877056. Current Snapshot: MvccSnapshot[applied={T|T < 7315589218546098176}]
I20260806 16:08:44.083310 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "014bd1a47b8043668331e000663f6810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "014bd1a47b8043668331e000663f6810" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43711 } } peers { permanent_uuid: "df5294b228a247c995937c01fa4b8e10" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34453 } } peers { permanent_uuid: "8b4b917f14e1439684d7c60f0fc7b9bf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 36513 } } }
I20260806 16:08:44.083446 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:44.088707 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:44.054838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.13.148.62:43711)
0806 16:08:44.054954 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.054958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.062730 (+ 7772us) server_negotiation.cc:407] Connection header received
0806 16:08:44.063393 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.063397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.063449 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.063505 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.064980 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.065849 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.071486 (+ 5637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.071882 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.071964 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.084076 (+ 12112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:44.084080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:44.084084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:44.084115 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:44.086280 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.086863 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.086867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.086869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.086964 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.087299 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.087302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.087304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.087532 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:44.087676 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:44.088375 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 16:08:44.088496 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 16:08:44.088928 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:44.055619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.13.148.60:36513)
0806 16:08:44.055719 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.055725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.064901 (+ 9176us) server_negotiation.cc:407] Connection header received
0806 16:08:44.068037 (+ 3136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.068040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.068096 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.068155 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.069384 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.070225 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.082647 (+ 12422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:44.083083 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:44.083174 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:44.085449 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:44.085452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:44.085455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:44.085479 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:44.087133 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.087613 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.087616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.087618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.087715 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:44.088000 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:44.088003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:44.088005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:44.088197 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:44.088296 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:44.088657 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 16:08:44.088770 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260806 16:08:44.090523 13365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:44.125756 13348 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:44.127198 13348 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:44.145364 13348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 014bd1a47b8043668331e000663f6810: Generated new TSK 0
I20260806 16:08:44.146312 13348 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:08:44.181137 13096 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.177964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.13.148.62:43711)
0806 16:08:44.178098 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.178103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.179935 (+ 1832us) server_negotiation.cc:407] Connection header received
0806 16:08:44.180146 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.180150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.180203 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.180261 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.180985 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59436: BlockingRecv error: recv got EOF from 127.0.0.1:59436 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 16:08:44.182734 13301 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.178772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.13.148.60:36513)
0806 16:08:44.178909 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.178914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.181697 (+ 2783us) server_negotiation.cc:407] Connection header received
0806 16:08:44.181885 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.181889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.181941 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.181997 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.182590 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38398: BlockingRecv error: recv got EOF from 127.0.0.1:38398 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 16:08:44.184064 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.179390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.13.148.61:34453)
0806 16:08:44.179516 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.179521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.183232 (+ 3711us) server_negotiation.cc:407] Connection header received
0806 16:08:44.183373 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.183377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.183433 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.183540 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.183938 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53520: BlockingRecv error: recv got EOF from 127.0.0.1:53520 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 16:08:44.188889 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.187205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.13.148.61:34453)
0806 16:08:44.187347 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.187351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.187901 (+ 550us) server_negotiation.cc:407] Connection header received
0806 16:08:44.188124 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.188128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.188205 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.188263 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.188790 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53524: BlockingRecv error: recv got EOF from 127.0.0.1:53524 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 16:08:44.195055 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.193350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.13.148.61:34453)
0806 16:08:44.193455 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.193459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.193988 (+ 529us) server_negotiation.cc:407] Connection header received
0806 16:08:44.194193 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.194197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.194251 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.194307 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.194954 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53530: BlockingRecv error: recv got EOF from 127.0.0.1:53530 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 16:08:44.201122 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.199498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.13.148.61:34453)
0806 16:08:44.199604 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.199608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.200119 (+ 511us) server_negotiation.cc:407] Connection header received
0806 16:08:44.200326 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.200332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.200388 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.200444 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.201010 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53544: BlockingRecv error: recv got EOF from 127.0.0.1:53544 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 16:08:44.209785 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.207882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.13.148.61:34453)
0806 16:08:44.208070 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.208075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.208682 (+ 607us) server_negotiation.cc:407] Connection header received
0806 16:08:44.208907 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.208912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.208978 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.209046 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.209649 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53550: BlockingRecv error: recv got EOF from 127.0.0.1:53550 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 16:08:44.222458 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.220555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.13.148.61:34453)
0806 16:08:44.220699 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.220704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.221319 (+ 615us) server_negotiation.cc:407] Connection header received
0806 16:08:44.221553 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.221558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.221622 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.221699 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.222334 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53556: BlockingRecv error: recv got EOF from 127.0.0.1:53556 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 16:08:44.243273 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.241380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.13.148.61:34453)
0806 16:08:44.241529 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.241535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.242112 (+ 577us) server_negotiation.cc:407] Connection header received
0806 16:08:44.242327 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.242331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.242388 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.242443 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.243145 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53568: BlockingRecv error: recv got EOF from 127.0.0.1:53568 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 16:08:44.279333 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.277888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.13.148.61:34453)
0806 16:08:44.278023 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.278029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.278315 (+ 286us) server_negotiation.cc:407] Connection header received
0806 16:08:44.278515 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.278523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.278574 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.278669 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.279175 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53580: BlockingRecv error: recv got EOF from 127.0.0.1:53580 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 16:08:44.347388 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.345855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.13.148.61:34453)
0806 16:08:44.346039 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.346045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.346199 (+ 154us) server_negotiation.cc:407] Connection header received
0806 16:08:44.346399 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.346403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.346538 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.346640 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.347202 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53590: BlockingRecv error: recv got EOF from 127.0.0.1:53590 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 16:08:44.479188 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.477844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.13.148.61:34453)
0806 16:08:44.478012 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.478019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.478139 (+ 120us) server_negotiation.cc:407] Connection header received
0806 16:08:44.478332 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.478335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.478391 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.478511 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.479058 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53596: BlockingRecv error: recv got EOF from 127.0.0.1:53596 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 16:08:44.738885 13202 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:44.737662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.13.148.61:34453)
0806 16:08:44.737816 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:44.737820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:44.737836 (+ 16us) server_negotiation.cc:407] Connection header received
0806 16:08:44.737928 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:44.737930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:44.737979 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:44.738070 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:44.738679 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53604: BlockingRecv error: recv got EOF from 127.0.0.1:53604 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 16:08:45.035576 13338 catalog_manager.cc:1259] Loaded cluster ID: 62407cfb29274de7a68ea6c827e35cab
I20260806 16:08:45.035681 13338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10: loading cluster ID for follower catalog manager: success
I20260806 16:08:45.037648 13338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10: acquiring CA information for follower catalog manager: success
I20260806 16:08:45.039136 13338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df5294b228a247c995937c01fa4b8e10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:08:45.091886 13365 catalog_manager.cc:1259] Loaded cluster ID: 62407cfb29274de7a68ea6c827e35cab
I20260806 16:08:45.092038 13365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf: loading cluster ID for follower catalog manager: success
I20260806 16:08:45.093942 13365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf: acquiring CA information for follower catalog manager: success
I20260806 16:08:45.095355 13365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b4b917f14e1439684d7c60f0fc7b9bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:08:45.256847 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:45.252941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.13.148.61:34453)
0806 16:08:45.253190 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:45.253194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:45.253268 (+ 74us) server_negotiation.cc:407] Connection header received
0806 16:08:45.253382 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:45.253384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:45.253515 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:45.253629 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:45.253884 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.254416 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.255337 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.255556 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.255636 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:45.255922 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:45.256379 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:45.256473 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:45.256595 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 16:08:45.256644 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
I20260806 16:08:45.265372 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:45.260763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.13.148.60:36513)
0806 16:08:45.261047 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:45.261051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:45.261077 (+ 26us) server_negotiation.cc:407] Connection header received
0806 16:08:45.261242 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:45.261244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:45.261363 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:45.261470 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:45.261904 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.262745 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.263728 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.263971 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.264181 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:45.264435 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:45.264907 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:45.265028 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:45.265176 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 16:08:45.265232 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":132,"threads_started":1}
I20260806 16:08:45.274501 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:45.269049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.13.148.61:34453)
0806 16:08:45.269212 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:45.269217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:45.269750 (+ 533us) server_negotiation.cc:407] Connection header received
0806 16:08:45.270119 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:45.270122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:45.270182 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:45.270282 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:45.270696 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.271276 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.272797 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.273198 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.273327 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:45.273848 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:45.273930 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:45.274025 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:45.274209 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 16:08:45.274284 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 16:08:45.274735 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:45.269867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.13.148.60:36513)
0806 16:08:45.270017 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:45.270022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:45.270035 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:08:45.270079 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:45.270082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:45.270130 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:45.270220 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:45.270545 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.271412 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.273416 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.273666 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.273801 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:45.274165 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:45.274248 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:45.274320 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:45.274507 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 16:08:45.274565 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 16:08:45.275194 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:45.269184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.13.148.62:43711)
0806 16:08:45.269845 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:45.269849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:45.269863 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:08:45.269989 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:45.269992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:45.270184 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:45.270705 (+ 521us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:45.270835 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.271674 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.272854 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:45.273241 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:45.273345 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:45.273858 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:45.274260 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:45.274354 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:45.274948 (+ 594us) server_negotiation.cc:299] Negotiation successful
0806 16:08:45.275024 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":76,"threads_started":1}
I20260806 16:08:45.284492 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 13008
I20260806 16:08:45.298622 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 13101
I20260806 16:08:45.310216 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 13203
2026-08-06T16:08:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3446 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 16:08:45 dist-test-slave-7kzw krb5kdc[13510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 16:08:45 dist-test-slave-7kzw krb5kdc[13510](info): set up 2 sockets
Aug 06 16:08:45 dist-test-slave-7kzw krb5kdc[13510](info): commencing operation
krb5kdc: starting...
W20260806 16:08:46.541375 13904 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/test-user.keytab.
Aug 06 16:08:46 dist-test-slave-7kzw krb5kdc[13510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T16:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T16:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260806 16:08:46.679697 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:38593
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:38593,127.13.148.61:41551,127.13.148.60:42481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 16:08:46.796000 13530 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:08:46.796322 13530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:46.796397 13530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:46.796474 13530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:46.800482 13530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:46.800565 13530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:46.800715 13530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:46.800829 13530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:46.800865 13530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:46.806315 13530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:38593,127.13.148.61:41551,127.13.148.60:42481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:46.807490 13530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:46.808573 13530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:46.817420 13536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:46.817420 13537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:46.817432 13541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:46.817854 13530 server_base.cc:1034] running on GCE node
I20260806 16:08:46.818418 13530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:46.819545 13530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:46.820719 13530 hybrid_clock.cc:648] HybridClock initialized: now 1786032526820690 us; error 41 us; skew 500 ppm
Aug 06 16:08:46 dist-test-slave-7kzw krb5kdc[13510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032526, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:46.823657 13530 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260806 16:08:46.824507 13530 webserver.cc:460] Webserver started at http://127.13.148.62:35223/ using document root <none> and password file <none>
I20260806 16:08:46.825065 13530 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:46.825157 13530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:46.825407 13530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:46.827512 13530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/data/instance:
uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6"
format_stamp: "Formatted at 2026-08-06 16:08:46 on dist-test-slave-7kzw"
server_key: "e7049eb2520e25009bff9222f8188f14"
server_key_iv: "8c355301bae2caa18729e093b1b8ca58"
server_key_version: "encryptionkey@0"
I20260806 16:08:46.828102 13530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/wal/instance:
uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6"
format_stamp: "Formatted at 2026-08-06 16:08:46 on dist-test-slave-7kzw"
server_key: "e7049eb2520e25009bff9222f8188f14"
server_key_iv: "8c355301bae2caa18729e093b1b8ca58"
server_key_version: "encryptionkey@0"
I20260806 16:08:46.831926 13530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 16:08:46.834384 13551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:46.835767 13530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 16:08:46.835933 13530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/wal
uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6"
format_stamp: "Formatted at 2026-08-06 16:08:46 on dist-test-slave-7kzw"
server_key: "e7049eb2520e25009bff9222f8188f14"
server_key_iv: "8c355301bae2caa18729e093b1b8ca58"
server_key_version: "encryptionkey@0"
I20260806 16:08:46.836053 13530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:46.856256 13530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:46.859745 13530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:46.859985 13530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:46.867401 13530 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:38593
I20260806 16:08:46.867692 13624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:38593 every 8 connection(s)
I20260806 16:08:46.868467 13530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/data/info.pb
I20260806 16:08:46.871609 13626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:46.875749 13626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:46.876878 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 13530
I20260806 16:08:46.877023 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-0/wal/instance
I20260806 16:08:46.877355 13904 external_mini_cluster.cc:1428] Setting key cd2eb49878240f2ab1d5b808d232a53e
Aug 06 16:08:46 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032526, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.61@KRBTEST.COM" created.
I20260806 16:08:46.892856 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:46.878443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38593 (local address 127.0.0.1:46796)
0806 16:08:46.879101 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:46.879139 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:46.879990 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:46.880691 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:46.880696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:46.881156 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:46.881395 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:46.881408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:46.882602 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:46.882605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:46.883316 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:46.883326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:46.883968 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:46.883972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:46.884105 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:46.884855 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:46.884872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:46.886679 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:46.889086 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:46.889095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:46.889104 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:46.889380 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:46.889846 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:46.889852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:46.889857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:46.890007 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:46.890940 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:46.890947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:46.891100 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 16:08:46.891307 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 16:08:46.891727 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":135,"spinlock_wait_cycles":39168,"thread_start_us":170,"threads_started":1}
I20260806 16:08:46.892856 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:46.878442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.13.148.62:38593)
0806 16:08:46.878940 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:46.878959 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:46.879248 (+ 289us) server_negotiation.cc:407] Connection header received
0806 16:08:46.880303 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:46.880308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:46.880494 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:46.880687 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:46.881550 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:46.882425 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:46.883442 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:46.883811 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:46.883993 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:46.886852 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:46.886857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:46.886867 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:46.886889 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:46.888965 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:46.889532 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:46.889537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:46.889538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:46.889664 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:46.890166 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:46.890169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:46.890171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:46.890808 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:46.890974 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:46.891376 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 16:08:46.891661 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":307,"thread_start_us":154,"threads_started":1}
I20260806 16:08:46.895496 13626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:46.896399 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:46.896026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52520)
0806 16:08:46.896192 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:46.896293 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 16:08:46.896679 13556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
I20260806 16:08:46.897713 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:46.897390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52534)
0806 16:08:46.897560 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:46.897623 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 16:08:46.897917 13626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:41551: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
I20260806 16:08:46.909957 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:41551
--webserver_interface=127.13.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:38593,127.13.148.61:41551,127.13.148.60:42481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:08:46.947206 13626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } attempt: 1
I20260806 16:08:46.948194 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:46.947843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52548)
0806 16:08:46.948028 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:46.948093 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 16:08:46.948982 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:46.948758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52564)
0806 16:08:46.948875 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:46.948927 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 16:08:46.949100 13626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:41551: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
I20260806 16:08:47.017280 13626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } attempt: 2
I20260806 16:08:47.018239 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.017898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52572)
0806 16:08:47.018090 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.018158 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 16:08:47.019069 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.018829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52584)
0806 16:08:47.018951 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.019006 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 16:08:47.019199 13626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:41551: Network error: Client connection negotiation failed: client connection to 127.13.148.61:41551: connect: Connection refused (error 111)
W20260806 16:08:47.026012 13634 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 16:08:47.026325 13634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:47.026388 13634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:47.026500 13634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 16:08:47.030450 13634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:47.030565 13634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:47.030613 13634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:47.030646 13634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:47.030678 13634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:47.035954 13634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:38593,127.13.148.61:41551,127.13.148.60:42481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:41551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:47.037015 13634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:47.038079 13634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 16:08:47.046720 13644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:47.046747 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:47.046768 13643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:47.046984 13634 server_base.cc:1034] running on GCE node
I20260806 16:08:47.047670 13634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:47.048806 13634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:47.049985 13634 hybrid_clock.cc:648] HybridClock initialized: now 1786032527049964 us; error 32 us; skew 500 ppm
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:47.053097 13634 init.cc:377] Logged in from keytab as kudu/127.13.148.61@KRBTEST.COM (short username kudu)
I20260806 16:08:47.053921 13634 webserver.cc:460] Webserver started at http://127.13.148.61:38321/ using document root <none> and password file <none>
I20260806 16:08:47.054440 13634 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:47.054580 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:47.054774 13634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:47.056751 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/data/instance:
uuid: "a4228455a183403cb53d0d0d81cb3956"
format_stamp: "Formatted at 2026-08-06 16:08:47 on dist-test-slave-7kzw"
server_key: "7f6d8f1110c175f874ecd0fc26c39853"
server_key_iv: "23d8a8d39e1ed1d1b0a29f8fa7415513"
server_key_version: "encryptionkey@0"
I20260806 16:08:47.057358 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/wal/instance:
uuid: "a4228455a183403cb53d0d0d81cb3956"
format_stamp: "Formatted at 2026-08-06 16:08:47 on dist-test-slave-7kzw"
server_key: "7f6d8f1110c175f874ecd0fc26c39853"
server_key_iv: "23d8a8d39e1ed1d1b0a29f8fa7415513"
server_key_version: "encryptionkey@0"
I20260806 16:08:47.061142 13634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 16:08:47.063786 13656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:47.065197 13634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 16:08:47.065315 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/wal
uuid: "a4228455a183403cb53d0d0d81cb3956"
format_stamp: "Formatted at 2026-08-06 16:08:47 on dist-test-slave-7kzw"
server_key: "7f6d8f1110c175f874ecd0fc26c39853"
server_key_iv: "23d8a8d39e1ed1d1b0a29f8fa7415513"
server_key_version: "encryptionkey@0"
I20260806 16:08:47.065470 13634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:47.077832 13634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:47.081475 13634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:47.081717 13634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:47.089395 13634 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:41551
I20260806 16:08:47.089686 13725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:41551 every 8 connection(s)
I20260806 16:08:47.090504 13634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/data/info.pb
I20260806 16:08:47.093334 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:47.097075 13726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:47.097975 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 13634
I20260806 16:08:47.098161 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-1/wal/instance
I20260806 16:08:47.098495 13904 external_mini_cluster.cc:1428] Setting key 5547a53b3aeb5fd25ec6fad60ce9b279
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.60@KRBTEST.COM; defaulting to no policy
I20260806 16:08:47.110373 13626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.60@KRBTEST.COM" created.
I20260806 16:08:47.114321 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.099123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.13.148.62:38593)
0806 16:08:47.099327 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.099335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.100098 (+ 763us) server_negotiation.cc:407] Connection header received
0806 16:08:47.101309 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.101313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.101366 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.101458 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.102953 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.103977 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.105110 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.105612 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.105772 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.108788 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.108796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.108803 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.108842 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.111541 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.112195 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.112199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.112201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.112297 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.112751 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.112755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.112756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.113005 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.113084 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.113860 (+ 776us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.114023 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 16:08:47.115186 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.099456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38593 (local address 127.0.0.1:46808)
0806 16:08:47.099926 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.099979 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:47.101059 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:47.101667 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:47.101679 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:47.102221 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:47.102793 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.102807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.104123 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.104126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.104973 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.104982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.105815 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.105818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.105952 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.106673 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:47.106690 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:47.108571 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:47.111646 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.111653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.111668 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.112063 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.112402 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.112405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.112407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.112613 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.113209 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:47.113216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:47.113514 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 16:08:47.113748 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 16:08:47.114099 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":144,"threads_started":1}
I20260806 16:08:47.116762 13726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032526, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:08:47.123001 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.110919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52588)
0806 16:08:47.111129 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.111146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:47.111232 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:47.111928 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:47.111931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:47.112070 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:47.112279 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.112288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.113846 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.113850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.114761 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.114777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.115762 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.115769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.115916 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.116458 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:47.116476 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:47.118176 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:47.121056 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.121060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.121063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.121287 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.121622 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.121627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.121629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.121733 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.122359 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:47.122362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:47.122511 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 16:08:47.122662 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 16:08:47.122812 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260806 16:08:47.123395 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.111030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.13.148.61:41551)
0806 16:08:47.111432 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.111443 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.111467 (+ 24us) server_negotiation.cc:407] Connection header received
0806 16:08:47.111505 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.111509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.111792 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.111961 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.112402 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.113685 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.114903 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.115592 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.116190 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.118329 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.118339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.118352 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.118374 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.120911 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.121412 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.121419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.121421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.121517 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.121836 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.121841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.121843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.122219 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.122387 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.122766 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.122915 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":189,"threads_started":1}
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:08:47.125402 13626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:47.126266 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.125895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34256)
0806 16:08:47.126104 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.126174 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 16:08:47.127079 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.126797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34266)
0806 16:08:47.126953 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.127010 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260806 16:08:47.127286 13626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42481: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
I20260806 16:08:47.127838 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.117333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52592)
0806 16:08:47.117525 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.117542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:47.117622 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:47.118267 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:47.118274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:47.118515 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:47.118678 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.118692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.119656 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.119664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.120738 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.120754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.121621 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.121624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.121750 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.122236 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:47.122252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:47.124207 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:47.126212 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.126215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.126216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.126467 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.126794 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.126799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.126802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.126897 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.127392 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:47.127395 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:47.127498 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 16:08:47.127602 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 16:08:47.127734 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 16:08:47.128033 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.117398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.13.148.61:41551)
0806 16:08:47.118000 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.118009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.118023 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:08:47.118063 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.118070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.118120 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.118227 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.118831 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.119530 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.120900 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.121382 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.121501 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.124368 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.124373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.124377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.124406 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.126093 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.126595 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.126598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.126599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.126689 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.127006 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.127010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.127012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.127284 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.127401 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.127692 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.127795 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":114,"threads_started":1}
I20260806 16:08:47.128657 13726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:47.129606 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.129299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34280)
0806 16:08:47.129482 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.129552 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260806 16:08:47.129874 13657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
I20260806 16:08:47.130796 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.130558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34284)
0806 16:08:47.130702 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.130747 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 16:08:47.130940 13726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42481: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
I20260806 16:08:47.134044 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
/tmp/dist-test-taskwTlxSD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:42481
--webserver_interface=127.13.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:38593,127.13.148.61:41551,127.13.148.60:42481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 16:08:47.158695 13626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } attempt: 1
I20260806 16:08:47.159651 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.159336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34294)
0806 16:08:47.159494 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.159559 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 16:08:47.160490 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.160232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34310)
0806 16:08:47.160370 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.160429 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 16:08:47.160722 13626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42481: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
I20260806 16:08:47.180214 13726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } attempt: 1
I20260806 16:08:47.181126 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.180787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34312)
0806 16:08:47.180954 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.181029 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 16:08:47.181917 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.181637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34324)
0806 16:08:47.181795 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.181859 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 16:08:47.182163 13726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42481: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
I20260806 16:08:47.235890 13626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } attempt: 2
I20260806 16:08:47.236857 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.236523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34340)
0806 16:08:47.236692 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.236770 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 16:08:47.237819 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.237508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34346)
0806 16:08:47.237680 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.237740 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 16:08:47.237977 13626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42481: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
I20260806 16:08:47.250402 13726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } attempt: 2
W20260806 16:08:47.251192 13737 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260806 16:08:47.251408 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.251088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34352)
0806 16:08:47.251263 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.251329 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260806 16:08:47.251487 13737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 16:08:47.251536 13737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 16:08:47.251597 13737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 16:08:47.252334 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.252050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34362)
0806 16:08:47.252204 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.252255 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 16:08:47.252682 13726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42481: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
W20260806 16:08:47.256134 13737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 16:08:47.256189 13737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 16:08:47.256210 13737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 16:08:47.256227 13737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 16:08:47.256284 13737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 16:08:47.261610 13737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:38593,127.13.148.61:41551,127.13.148.60:42481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:42481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 15:06:40 UTC on ccycloud.quasar-amrkwm.root.comops.site
build id 4570627
I20260806 16:08:47.262712 13737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 16:08:47.263722 13737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 16:08:47.273003 13737 server_base.cc:1034] running on GCE node
W20260806 16:08:47.272943 13743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:47.272965 13742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 16:08:47.272957 13748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 16:08:47.274006 13737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 16:08:47.275168 13737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 16:08:47.276355 13737 hybrid_clock.cc:648] HybridClock initialized: now 1786032527276341 us; error 25 us; skew 500 ppm
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 16:08:47.279412 13737 init.cc:377] Logged in from keytab as kudu/127.13.148.60@KRBTEST.COM (short username kudu)
I20260806 16:08:47.280388 13737 webserver.cc:460] Webserver started at http://127.13.148.60:44781/ using document root <none> and password file <none>
I20260806 16:08:47.281057 13737 fs_manager.cc:359] Metadata directory not provided
I20260806 16:08:47.281174 13737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 16:08:47.281428 13737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 16:08:47.283820 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/data/instance:
uuid: "727ae354d83e4c85ac25996c88df5b4f"
format_stamp: "Formatted at 2026-08-06 16:08:47 on dist-test-slave-7kzw"
server_key: "fcc7e8e30b7ba8d7a8943348b0553a8e"
server_key_iv: "6fe143fea8733dd81bfca05c4ce652d0"
server_key_version: "encryptionkey@0"
I20260806 16:08:47.284456 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/wal/instance:
uuid: "727ae354d83e4c85ac25996c88df5b4f"
format_stamp: "Formatted at 2026-08-06 16:08:47 on dist-test-slave-7kzw"
server_key: "fcc7e8e30b7ba8d7a8943348b0553a8e"
server_key_iv: "6fe143fea8733dd81bfca05c4ce652d0"
server_key_version: "encryptionkey@0"
I20260806 16:08:47.288481 13737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 16:08:47.291013 13756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 16:08:47.292302 13737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 16:08:47.292469 13737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/wal
uuid: "727ae354d83e4c85ac25996c88df5b4f"
format_stamp: "Formatted at 2026-08-06 16:08:47 on dist-test-slave-7kzw"
server_key: "fcc7e8e30b7ba8d7a8943348b0553a8e"
server_key_iv: "6fe143fea8733dd81bfca05c4ce652d0"
server_key_version: "encryptionkey@0"
I20260806 16:08:47.292589 13737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 16:08:47.331514 13737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 16:08:47.335245 13737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 16:08:47.335493 13737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 16:08:47.337209 13626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } attempt: 3
I20260806 16:08:47.338105 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.337789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34366)
0806 16:08:47.337963 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.338020 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 16:08:47.338949 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.338713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34382)
0806 16:08:47.338838 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.338892 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 16:08:47.339183 13626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42481: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42481: connect: Connection refused (error 111)
I20260806 16:08:47.343964 13737 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:42481
I20260806 16:08:47.343904 13726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } attempt: 3
I20260806 16:08:47.344254 13825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:42481 every 8 connection(s)
I20260806 16:08:47.345283 13737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/data/info.pb
I20260806 16:08:47.348728 13831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 16:08:47.352684 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu as pid 13737
I20260806 16:08:47.352837 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwTlxSD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786032227601881-13904-0/minicluster-data/master-2/wal/instance
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:08:47.353158 13904 external_mini_cluster.cc:1428] Setting key d6edc2c9215182fd82be19629a7f10a4
I20260806 16:08:47.353427 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:47.358803 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.344366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34384)
0806 16:08:47.344521 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.344538 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:47.344641 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:47.347698 (+ 3057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:47.347701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:47.347877 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:47.347976 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.347981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.349728 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.349731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.350247 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.350256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.351233 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.351237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.351317 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.351886 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:47.351899 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:47.353424 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:47.356649 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.356655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.356658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.356942 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.357337 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.357340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.357342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.357437 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.358181 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:47.358184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:47.358295 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 16:08:47.358387 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 16:08:47.358575 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:08:47.361343 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.345297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.13.148.60:42481)
0806 16:08:47.345877 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.345898 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.345943 (+ 45us) server_negotiation.cc:407] Connection header received
0806 16:08:47.346841 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.346867 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.347316 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.347695 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.348102 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.349572 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.350372 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.351077 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.351244 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.353640 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.353655 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.353671 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.353702 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.356495 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.357070 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.357076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.357083 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.357220 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.357537 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.357541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.357543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.358059 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.359912 (+ 1853us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.360046 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.360317 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":161,"threads_started":1}
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260806 16:08:47.365945 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.353861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.13.148.60:42481)
0806 16:08:47.354582 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.354590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.354601 (+ 11us) server_negotiation.cc:407] Connection header received
0806 16:08:47.354630 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.354632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.354667 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.354774 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.355716 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.356485 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.357714 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.358036 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.358172 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.361146 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.361150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.361153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.361178 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.363923 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.364466 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.364470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.364472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.364577 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.364954 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.364958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.364961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.365196 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.365393 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.365597 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.365731 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":145,"threads_started":1}
I20260806 16:08:47.366408 13726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: Bootstrap starting.
I20260806 16:08:47.366974 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.356640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.13.148.62:38593)
0806 16:08:47.356805 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.356810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.356824 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:08:47.356872 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.356875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.356924 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.357013 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.357834 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.358490 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.359837 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.360184 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.360312 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.362731 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.362735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.362737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.362772 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.364602 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.365246 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.365252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.365256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.365357 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.365720 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.365723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.365725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.365981 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.366073 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.366708 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.366831 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:08:47.367326 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.355815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38593 (local address 127.0.0.1:46812)
0806 16:08:47.356195 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.356257 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:47.356372 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:47.357071 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:47.357078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:47.357493 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:47.357638 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.357662 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.358614 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.358617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.359695 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.359709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.360320 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.360323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.360427 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.360944 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:47.360958 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:47.362580 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:47.364735 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.364744 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.364750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.365060 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.365464 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.365467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.365471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.365592 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.366303 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:47.366309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:47.366455 (+ 146us) client_negotiation.cc:769] Sending connection context
0806 16:08:47.366623 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 16:08:47.366838 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
I20260806 16:08:47.368371 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:47.370076 13726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:47.371045 13726 log.cc:824] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: Log is configured to *not* fsync() on all Append() calls
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260806 16:08:47.374214 13726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: No bootstrap required, opened a new log
I20260806 16:08:47.377061 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.367434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.13.148.62:38593)
0806 16:08:47.367550 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.367554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.367627 (+ 73us) server_negotiation.cc:407] Connection header received
0806 16:08:47.367776 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.367779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.367824 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.367899 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.368674 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.369436 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.370506 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.370813 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.370921 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.373681 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.373687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.373690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.373719 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.375227 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.375663 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.375667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.375669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.375752 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.376051 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.376055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.376057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.376315 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.376539 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.376759 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.376879 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:08:47.379225 13726 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } }
I20260806 16:08:47.379506 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.368881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:41551 (local address 127.0.0.1:52608)
0806 16:08:47.369037 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.369050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:47.369125 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:47.369354 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:47.369357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:47.369555 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:47.369671 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.369678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.370739 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.370742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.371783 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.371797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.372487 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.372490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.372564 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.373064 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:47.373076 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:47.374761 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:47.377395 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.377399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.377401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.377637 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.377975 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.377979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.377981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.378441 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.378974 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:47.378977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:47.379078 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 16:08:47.379200 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 16:08:47.379345 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 16:08:47.379492 13726 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4228455a183403cb53d0d0d81cb3956, State: Initialized, Role: FOLLOWER
I20260806 16:08:47.379829 13726 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } }
I20260806 16:08:47.380016 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.368940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.13.148.61:41551)
0806 16:08:47.369088 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.369092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.369107 (+ 15us) server_negotiation.cc:407] Connection header received
0806 16:08:47.369203 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.369206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.369246 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.369340 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.369806 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.370620 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.371932 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.372342 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.372481 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.374912 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.374916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.374919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.374943 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.377268 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.377747 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.377750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.377752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.377865 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.378574 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.378579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.378582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.378862 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.378960 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.379289 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.379409 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":72}
I20260806 16:08:47.380664 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } has no permanent_uuid. Determining permanent_uuid...
I20260806 16:08:47.381393 13726 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
W20260806 16:08:47.387415 13859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:47.387485 13859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:47.385286 13634 master_runner.cc:186] Error getting master registration for 127.13.148.62:38593: OK, instance_id { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" instance_seqno: 1786032526865303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:08:47.388613 13634 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:08:47.389513 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.378332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.13.148.61:41551)
0806 16:08:47.378441 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.378447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.378460 (+ 13us) server_negotiation.cc:407] Connection header received
0806 16:08:47.378635 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.378640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.378685 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.378735 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.379586 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.380338 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.381210 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.381561 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.381634 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.384758 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.384762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.384764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.384789 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.387314 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.387877 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.387883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.387886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.387986 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.388369 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.388373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.388376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.388639 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.388827 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.389242 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.389366 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:08:47.391016 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.381174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34398)
0806 16:08:47.381279 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.381293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:47.381365 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:47.381610 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:47.381613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:47.381791 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:47.381887 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.381893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.383068 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.383073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.383917 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.383931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.384649 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.384655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.384748 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.385471 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:47.385490 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:47.387350 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:47.389350 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.389353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.389354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.389596 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.389880 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.389882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.389884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.389973 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.390488 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:47.390492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:47.390590 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 16:08:47.390715 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 16:08:47.390851 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 16:08:47.391415 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.381325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.13.148.60:42481)
0806 16:08:47.381422 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.381426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.381438 (+ 12us) server_negotiation.cc:407] Connection header received
0806 16:08:47.381469 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.381472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.381512 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.381607 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.382023 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.382932 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.384055 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.384432 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.384538 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.387474 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.387477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.387481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.387509 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.389242 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.389698 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.389702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.389706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.389786 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.390071 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.390077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.390079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.390346 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.390449 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.391152 (+ 703us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.391274 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 16:08:47.393296 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 16:08:47.394191 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f: Bootstrap starting.
I20260806 16:08:47.394711 13843 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.395366 13843 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:47.394721 13592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46828
I20260806 16:08:47.397127 13831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:47.397989 13831 log.cc:824] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:47.399948 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f: No bootstrap required, opened a new log
I20260806 16:08:47.404121 13831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } }
I20260806 16:08:47.404382 13831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727ae354d83e4c85ac25996c88df5b4f, State: Initialized, Role: FOLLOWER
I20260806 16:08:47.404690 13831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } }
I20260806 16:08:47.405184 13863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.405318 13863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:47.405941 13831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [sys.catalog]: configured and running, proceeding with master startup.
W20260806 16:08:47.411440 13879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 16:08:47.411535 13879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 16:08:47.410082 13737 master_runner.cc:186] Error getting master registration for 127.13.148.62:38593: OK, instance_id { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" instance_seqno: 1786032526865303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 16:08:47.413345 13737 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:08:47.503309 13626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } attempt: 4
Aug 06 16:08:47 dist-test-slave-7kzw krb5kdc[13510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032526, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260806 16:08:47.514035 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.503956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42481 (local address 127.0.0.1:34410)
0806 16:08:47.504310 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 16:08:47.504326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 16:08:47.504429 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 16:08:47.504722 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 16:08:47.504726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 16:08:47.504966 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 16:08:47.505106 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.505114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.506154 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.506158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.507036 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 16:08:47.507048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.507696 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.507700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 16:08:47.507800 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.508420 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 16:08:47.508436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 16:08:47.510177 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 16:08:47.512322 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.512326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.512331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.512606 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.512922 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 16:08:47.512928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 16:08:47.512931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 16:08:47.513041 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 16:08:47.513478 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 16:08:47.513483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 16:08:47.513590 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 16:08:47.513706 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 16:08:47.513852 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260806 16:08:47.514078 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.504308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.13.148.60:42481)
0806 16:08:47.504442 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.504447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.504505 (+ 58us) server_negotiation.cc:407] Connection header received
0806 16:08:47.504557 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.504560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.504610 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.504694 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.505251 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.506050 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.507190 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.507568 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.507671 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.510321 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.510325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.510328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.510355 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.512189 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.512720 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.512723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.512725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.512819 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.513155 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.513159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.513161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.513380 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.513479 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.513784 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.513914 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 16:08:47.517956 13626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6: Bootstrap starting.
I20260806 16:08:47.520882 13626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6: Neither blocks nor log segments found. Creating new log.
I20260806 16:08:47.521855 13626 log.cc:824] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6: Log is configured to *not* fsync() on all Append() calls
I20260806 16:08:47.524281 13626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6: No bootstrap required, opened a new log
I20260806 16:08:47.527603 13626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } }
I20260806 16:08:47.527921 13626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03dc2a38a6db461ab0ad4095e4eeb7f6, State: Initialized, Role: FOLLOWER
I20260806 16:08:47.528348 13626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } }
I20260806 16:08:47.529852 13626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 16:08:47.529973 13592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 16:08:47.530129 13592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:47.531539 13592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } }
I20260806 16:08:47.532466 13882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.532639 13882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:47.533699 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a4228455a183403cb53d0d0d81cb3956"
I20260806 16:08:47.533969 13694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:47.535462 13694 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03dc2a38a6db461ab0ad4095e4eeb7f6 in term 1.
I20260806 16:08:47.535964 13556 leader_election.cc:302] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03dc2a38a6db461ab0ad4095e4eeb7f6, a4228455a183403cb53d0d0d81cb3956; no voters:
I20260806 16:08:47.536409 13882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 16:08:47.536706 13882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [term 1 LEADER]: Becoming Leader. State: Replica: 03dc2a38a6db461ab0ad4095e4eeb7f6, State: Running, Role: LEADER
I20260806 16:08:47.537130 13882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } }
I20260806 16:08:47.539302 13882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03dc2a38a6db461ab0ad4095e4eeb7f6. Latest consensus state: current_term: 1 leader_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.539562 13882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:47.540334 13901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 16:08:47.541006 13530 master_runner.cc:232] Existing masters have differing terms: 1 vs 0
I20260806 16:08:47.541131 13530 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 16:08:47.541733 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "727ae354d83e4c85ac25996c88df5b4f"
I20260806 16:08:47.541967 13791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [term 0 FOLLOWER]: Advancing to term 1
I20260806 16:08:47.543408 13791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03dc2a38a6db461ab0ad4095e4eeb7f6 in term 1.
I20260806 16:08:47.543830 13592 leader_election.cc:288] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [CANDIDATE]: Term 1 election: Requested vote from peers a4228455a183403cb53d0d0d81cb3956 (127.13.148.61:41551), 727ae354d83e4c85ac25996c88df5b4f (127.13.148.60:42481)
I20260806 16:08:47.545620 13901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 16:08:47.564158 13791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [term 1 FOLLOWER]: Refusing update from remote peer 03dc2a38a6db461ab0ad4095e4eeb7f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:47.565037 13899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:47.569630 13863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03dc2a38a6db461ab0ad4095e4eeb7f6. Latest consensus state: current_term: 1 leader_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.569759 13863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:47.572794 13694 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [term 1 FOLLOWER]: Refusing update from remote peer 03dc2a38a6db461ab0ad4095e4eeb7f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 16:08:47.573427 13899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 16:08:47.579063 13899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.579214 13899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:47.579561 13843 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03dc2a38a6db461ab0ad4095e4eeb7f6. Latest consensus state: current_term: 1 leader_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.579690 13843 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:47.580675 13863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.580790 13863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f [sys.catalog]: This master's current role is: FOLLOWER
W20260806 16:08:47.583444 13664 tablet.cc:2369] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 16:08:47.583508 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.546338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.13.148.62:38593)
0806 16:08:47.546455 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.546459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.548524 (+ 2065us) server_negotiation.cc:407] Connection header received
0806 16:08:47.548734 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.548738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.548789 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.548844 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.549930 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.550802 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.552192 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.552522 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.552605 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.571355 (+ 18750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.571360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.571362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.571391 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.574843 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.581561 (+ 6718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.581566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.581569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.581681 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.582098 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.582101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.582103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.582349 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.582516 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.583225 (+ 709us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.583355 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:08:47.585242 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.547072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.13.148.60:42481)
0806 16:08:47.547190 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.547194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.553300 (+ 6106us) server_negotiation.cc:407] Connection header received
0806 16:08:47.553493 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.553497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.553549 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.553603 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.554623 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.555363 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.556494 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.556807 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.556887 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.564978 (+ 8091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.564982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.564985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.565013 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.566824 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.569720 (+ 2896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.569724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.569726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.569815 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.570485 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.570489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.570490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.570716 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.570835 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.584950 (+ 14115us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.585082 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 16:08:47.585906 13901 catalog_manager.cc:1347] Generated new cluster ID: a6b18c439cbd47a1a899b7c7db620516
I20260806 16:08:47.585973 13901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 16:08:47.586071 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:47.547800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.13.148.61:41551)
0806 16:08:47.547916 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.547920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.557759 (+ 9839us) server_negotiation.cc:407] Connection header received
0806 16:08:47.557952 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.557956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.558010 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.558065 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.559126 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.559865 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.560982 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:47.561336 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:47.561417 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:47.567394 (+ 5977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 16:08:47.567398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 16:08:47.567401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 16:08:47.567429 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 16:08:47.569208 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.570096 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.570099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.570100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.570194 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 16:08:47.584126 (+ 13932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 16:08:47.584130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 16:08:47.584132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 16:08:47.584410 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 16:08:47.584567 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:47.585796 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0806 16:08:47.585924 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 16:08:47.591132 13882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.591252 13882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6 [sys.catalog]: This master's current role is: LEADER
I20260806 16:08:47.592307 13843 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03dc2a38a6db461ab0ad4095e4eeb7f6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38593 } } peers { permanent_uuid: "a4228455a183403cb53d0d0d81cb3956" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 41551 } } peers { permanent_uuid: "727ae354d83e4c85ac25996c88df5b4f" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42481 } } }
I20260806 16:08:47.592412 13843 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 16:08:47.621255 13901 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 16:08:47.622578 13901 catalog_manager.cc:1504] Loading token signing keys...
I20260806 16:08:47.630082 13901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03dc2a38a6db461ab0ad4095e4eeb7f6: Generated new TSK 0
I20260806 16:08:47.631068 13901 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 16:08:47.654050 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.652372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.13.148.61:41551)
0806 16:08:47.652540 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.652556 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.652820 (+ 264us) server_negotiation.cc:407] Connection header received
0806 16:08:47.653031 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.653034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.653099 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.653154 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.653896 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52644: BlockingRecv error: recv got EOF from 127.0.0.1:52644 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260806 16:08:47.654742 13628 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.652409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.13.148.62:38593)
0806 16:08:47.652535 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.652540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.653454 (+ 914us) server_negotiation.cc:407] Connection header received
0806 16:08:47.653623 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.653628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.653691 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.653782 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.654583 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46848: BlockingRecv error: recv got EOF from 127.0.0.1:46848 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 16:08:47.654984 13838 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.652934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.13.148.60:42481)
0806 16:08:47.653157 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.653162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.653824 (+ 662us) server_negotiation.cc:407] Connection header received
0806 16:08:47.654050 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.654055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.654118 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.654184 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.654861 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34426: BlockingRecv error: recv got EOF from 127.0.0.1:34426 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260806 16:08:47.658258 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.657085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.13.148.61:41551)
0806 16:08:47.657213 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.657216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.657298 (+ 82us) server_negotiation.cc:407] Connection header received
0806 16:08:47.657449 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.657451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.657564 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.657636 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.658130 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52646: BlockingRecv error: recv got EOF from 127.0.0.1:52646 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 16:08:47.663868 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.661595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.13.148.61:41551)
0806 16:08:47.661735 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.661739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.662795 (+ 1056us) server_negotiation.cc:407] Connection header received
0806 16:08:47.662955 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.662961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.663061 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.663219 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.663685 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52650: BlockingRecv error: recv got EOF from 127.0.0.1:52650 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 16:08:47.676254 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.675299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.13.148.61:41551)
0806 16:08:47.675402 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.675406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.675507 (+ 101us) server_negotiation.cc:407] Connection header received
0806 16:08:47.675611 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.675614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.675669 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.675758 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.676140 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52652: BlockingRecv error: recv got EOF from 127.0.0.1:52652 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 16:08:47.683357 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.681878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.13.148.61:41551)
0806 16:08:47.682057 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.682061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.682437 (+ 376us) server_negotiation.cc:407] Connection header received
0806 16:08:47.682586 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.682589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.682644 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.682724 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.683250 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52662: BlockingRecv error: recv got EOF from 127.0.0.1:52662 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260806 16:08:47.693928 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.692872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.13.148.61:41551)
0806 16:08:47.692982 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.692985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.693208 (+ 223us) server_negotiation.cc:407] Connection header received
0806 16:08:47.693292 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.693295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.693350 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.693437 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.693819 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52664: BlockingRecv error: recv got EOF from 127.0.0.1:52664 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 16:08:47.713361 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.712152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.13.148.61:41551)
0806 16:08:47.712296 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.712300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.712458 (+ 158us) server_negotiation.cc:407] Connection header received
0806 16:08:47.712560 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.712563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.712627 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.712738 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.713234 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52672: BlockingRecv error: recv got EOF from 127.0.0.1:52672 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 16:08:47.749123 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.747653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.13.148.61:41551)
0806 16:08:47.747833 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.747837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.747975 (+ 138us) server_negotiation.cc:407] Connection header received
0806 16:08:47.748111 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.748113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.748170 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.748263 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.748970 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52688: BlockingRecv error: recv got EOF from 127.0.0.1:52688 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 16:08:47.816555 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.815317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.13.148.61:41551)
0806 16:08:47.815489 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.815492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.815634 (+ 142us) server_negotiation.cc:407] Connection header received
0806 16:08:47.815832 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.815835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.815903 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.816009 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.816443 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52696: BlockingRecv error: recv got EOF from 127.0.0.1:52696 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 16:08:47.948122 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:47.946861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.13.148.61:41551)
0806 16:08:47.947142 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:47.947146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:47.947163 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:08:47.947220 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:47.947222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:47.947286 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:47.947361 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:47.947955 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52712: BlockingRecv error: recv got EOF from 127.0.0.1:52712 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
W20260806 16:08:48.207808 13730 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 16:08:48.206370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.13.148.61:41551)
0806 16:08:48.206606 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:48.206610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:48.206694 (+ 84us) server_negotiation.cc:407] Connection header received
0806 16:08:48.206801 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:48.206804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:48.206890 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:48.206993 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 16:08:48.207664 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52722: BlockingRecv error: recv got EOF from 127.0.0.1:52722 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260806 16:08:48.388839 13859 catalog_manager.cc:1259] Loaded cluster ID: a6b18c439cbd47a1a899b7c7db620516
I20260806 16:08:48.388933 13859 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: loading cluster ID for follower catalog manager: success
I20260806 16:08:48.390658 13859 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: acquiring CA information for follower catalog manager: success
I20260806 16:08:48.391783 13859 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4228455a183403cb53d0d0d81cb3956: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:08:48.412891 13879 catalog_manager.cc:1259] Loaded cluster ID: a6b18c439cbd47a1a899b7c7db620516
I20260806 16:08:48.413019 13879 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f: loading cluster ID for follower catalog manager: success
I20260806 16:08:48.414958 13879 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f: acquiring CA information for follower catalog manager: success
I20260806 16:08:48.416218 13879 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 727ae354d83e4c85ac25996c88df5b4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 16:08:48.726007 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:48.722061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.13.148.61:41551)
0806 16:08:48.722356 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:48.722360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:48.722403 (+ 43us) server_negotiation.cc:407] Connection header received
0806 16:08:48.722457 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:48.722459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:48.722615 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:48.722705 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:48.723024 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.723584 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.724456 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.724688 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.724774 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:48.725109 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:48.725577 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:48.725685 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:48.725740 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 16:08:48.725792 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":113,"threads_started":1}
I20260806 16:08:48.733212 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:48.729368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.13.148.60:42481)
0806 16:08:48.729655 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:48.729663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:48.729680 (+ 17us) server_negotiation.cc:407] Connection header received
0806 16:08:48.729777 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:48.729780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:48.729903 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:48.730014 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:48.730257 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.730940 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.731810 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.732050 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.732142 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:48.732425 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:48.732875 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:48.732973 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:48.733032 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 16:08:48.733081 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260806 16:08:48.740635 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:48.736713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.13.148.60:42481)
0806 16:08:48.736826 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:48.736836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:48.737080 (+ 244us) server_negotiation.cc:407] Connection header received
0806 16:08:48.737295 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:48.737299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:48.737360 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:48.737428 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:48.737754 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.738244 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.739524 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.739835 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.739920 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:48.740193 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:48.740273 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:48.740338 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:48.740425 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 16:08:48.740482 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 16:08:48.741366 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:48.736301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.13.148.62:38593)
0806 16:08:48.736954 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:48.736959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:48.736973 (+ 14us) server_negotiation.cc:407] Connection header received
0806 16:08:48.737079 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:48.737082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:48.737269 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:48.737400 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:48.737647 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.738526 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.739627 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.739941 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.740045 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:48.740439 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:48.740813 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:48.740942 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:48.741121 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 16:08:48.741184 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":86,"threads_started":1}
I20260806 16:08:48.742115 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 16:08:48.736326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.13.148.61:41551)
0806 16:08:48.736463 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 16:08:48.736468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 16:08:48.736674 (+ 206us) server_negotiation.cc:407] Connection header received
0806 16:08:48.736920 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 16:08:48.736923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 16:08:48.736972 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 16:08:48.737066 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 16:08:48.737347 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.738222 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.739984 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 16:08:48.740428 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 16:08:48.740530 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 16:08:48.741404 (+ 874us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 16:08:48.741495 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 16:08:48.741580 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 16:08:48.741910 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 16:08:48.741973 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 16:08:48.744804 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 13530
I20260806 16:08:48.757853 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 13634
I20260806 16:08:48.767539 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwTlxSD/build/debug/bin/kudu with pid 13737
2026-08-06T16:08:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3459 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6906 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (301163 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST